File system enhancement

General discussion forum for Parsix GNU/Linux in English.

Moderators: aysabzevar, mandog, alan, bkashaf, xet7, cmost, blix

File system enhancement

Postby WickedHangover » Fri Jul 12, 2013 12:50 pm

I've been having conversations with my programming group about the processes we run against a digital preservation repository. Typically we write software to calculate checksums on files after they are stored on file systems. The process is fast for small files but typically the files we preserve are larger than 100MB and many times 10-100GB.

The purpose of the checksums is to ensure files have not degraded and we recalculate the checksums at different intervals depending on the importance of the files.

Here's the question I posed to my group. Rather than write code to handle checksums, why can't the operating system calculate them at the time of storing the file and then recalculate every time the file is changed? I'm using the date stamps attached to files as the example of other information I'd like the operating system to automatically provide.

The first thought was to use FUSE and extend the operating system. I think this is a reasonable path but it still represents a certain amount of overhead and would rather see the checksums performed at a level closer to the kernel.

I'm wondering if anyone has ever considered this.

The significance of the enhancement for libraries would be pretty big. To put this into a library perspective, in the coming year I will be ingesting 2.8mil digital objects into a repository. During the year 3 full months will be required just for the calculation of the checksums. If the OS handled this without the extra step in the procedural code stack, we could increase our throughput into the repository significantly. My projection would allow almost 750,000 additional objects to be ingested.
WickedHangover
 
Posts: 1
Joined: Fri Jul 12, 2013 12:42 pm

Re: File system enhancement

Postby andy » Thu Sep 12, 2013 11:52 am

Thank you for sharing to us.there are many person searching about that now they will find enough resources by your post.I would like to join your forum anyway so please continue sharing with us
:roll:
andy
 
Posts: 1
Joined: Thu Sep 12, 2013 11:50 am


Return to General Discussions - English

Who is online

Users browsing this forum: Yahoo [Bot] and 1 guest

cron