Skip to content

Commit

Permalink
Merge pull request #205 from matthewrmshin/fcm-make-flexible-checksum…
Browse files Browse the repository at this point in the history
…-method

fcm make: build: flexible checksum method
  • Loading branch information
benfitzpatrick committed Oct 12, 2015
2 parents 3c6edb2 + 60abbbc commit 47bfbb3
Show file tree
Hide file tree
Showing 9 changed files with 137 additions and 23 deletions.
7 changes: 7 additions & 0 deletions doc/user_guide/annex_cfg.html
Original file line number Diff line number Diff line change
Expand Up @@ -1058,6 +1058,13 @@ <h3 id="make.build">FCM Make Configuration: Build</h3>
<dd>The C linker will add each directory in this setting as a library
search path.</dd>

<dt id="make.build.prop.checksum-method">checksum-method</dt>

<dd>The checksum method to use to determine if the content of a file is
changed or not. Either MD5 or one of the SHA algorithms supported by the
Perl module <a href="http://perldoc.perl.org/Digest/SHA.html">Digest::SHA</a>
can be used. (default=<samp>md5</samp>)</dd>

<dt id="make.build.prop.cxx">cxx</dt>

<dd>The C++ compiler and linker. (default=<samp>g++</samp>)</dd>
Expand Down
18 changes: 15 additions & 3 deletions doc/user_guide/make.html
Original file line number Diff line number Diff line change
Expand Up @@ -1874,7 +1874,7 @@ <h3 id="build.target-update">Build Targets Update in Incremental Mode</h3>
A target is considered out of date if:</p>

<ul>
<li>the source file's MD5 checksum is changed.</li>
<li>the source file's checksum is changed.</li>

<li>a required property is modified.</li>

Expand All @@ -1892,10 +1892,10 @@ <h3 id="build.target-update">Build Targets Update in Incremental Mode</h3>
<li>a dependency is passing on a <q>modified</q> status for a dependency
type, which cannot be passed on by the target.</li>

<li>the target does not exist or its MD5 checksum is changed.</li>
<li>the target does not exist or its checksum is changed.</li>
</ul>

<p>If, after an update, the target's MD5 checksum is the same as before, the
<p>If, after an update, the target's checksum is the same as before, the
target will be considered unchanged and up to date. In an incremental build,
the use of checksum ensures that any targets manually modified by the user
after the previous build is rebuilt accordingly. It also prevents unnecessary
Expand All @@ -1913,6 +1913,18 @@ <h3 id="build.target-update">Build Targets Update in Incremental Mode</h3>
re-link the executable. This allows incremental builds to be more
efficient.</p>

<div class="well">
<p><strong><i class="icon-pencil"></i> Note - checksum algorithm</strong></p>

<p>By default, the MD5 algorithm is used to calculate the checksum. This is
normally good enough to detect whether a file is modified or not. If this is
insufficient for whatever reasons, you can tell the build system to use one
of the SHA algorithms supported by the Perl module <a href=
"http://perldoc.perl.org/Digest/SHA.html">Digest::SHA</a>, by setting the
value of <a href=
"annex_cfg.html#make.build.prop.checksum-method">build.prop{checksum-method}</a>.</p>
</div>

<h3 id="build.inherit">Build Inheritance</h3>

<p>If a previous build with a similar configuration exists in another
Expand Down
6 changes: 3 additions & 3 deletions doc/user_guide/overview.html
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,9 @@ <h2 id="build-and-extract">Build and Extract</h2>
<ul>
<li>Parallel build.</li>

<li>Efficient incremental build. Changes to the MD5 checksums of source
files and/or the build configuration (e.g. changes to the compiler
flags) trigger the appropriate re-compilation.</li>
<li>Efficient incremental build. Changes to the checksums of source files
and/or the build configuration (e.g. changes to the compiler flags) trigger
the appropriate re-compilation.</li>

<li>Inheritance of items from an existing build.</li>

Expand Down
4 changes: 2 additions & 2 deletions lib/FCM/Context/Make/Build.pm
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ following attributes:
=item checksum
The MD5 checksum of the source file.
The checksum of the source file.
=item deps
Expand Down Expand Up @@ -289,7 +289,7 @@ The target category, e.g. bin, etc, include, lib, o, src
=item checksum
The MD5 checksum of the target.
The checksum of the target.
=item deps
Expand Down
20 changes: 13 additions & 7 deletions lib/FCM/System/Make/Build.pm
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ our %CONFIG_PARSER_OF = (
our %PROP_OF = (
# [default , ns-ok]
'archive-ok-target-category' => [q{include o} , undef],
'checksum-method' => [q{} , undef],
'ignore-missing-dep-ns' => [q{} , undef],
'no-step-source' => [q{} , undef],
'no-inherit-source' => [q{} , undef],
Expand Down Expand Up @@ -554,13 +555,16 @@ sub _sources_type {
sub _sources_analyse {
my ($attrib_ref, $m_ctx, $ctx) = @_;
my $timer = $UTIL->timer();
my $checksum_method = _prop($attrib_ref, 'checksum-method', $ctx);
my %FILE_TYPE_UTIL_OF = %{$attrib_ref->{file_type_util_of}};
# Checksum
while (my ($ns, $source) = each(%{$ctx->get_source_of()})) {
if ( exists($FILE_TYPE_UTIL_OF{$source->get_type()})
&& !defined($source->get_checksum())
) {
$source->set_checksum($UTIL->file_md5($source->get_path()));
$source->set_checksum(
$UTIL->file_checksum($source->get_path(), $checksum_method),
);
}
}
# Source information
Expand Down Expand Up @@ -705,11 +709,12 @@ sub _targets_update {
,
);
# Performs targets update
my $checksum_method = _prop($attrib_ref, 'checksum-method', $ctx);
my %stat_of = ();
eval {
my $n_jobs = $m_ctx->get_option_of('jobs');
my $runner = $UTIL->task_runner(
sub {_target_update($attrib_ref, @_)},
sub {_target_update($attrib_ref, $checksum_method, @_)},
$n_jobs,
);
eval {
Expand Down Expand Up @@ -768,7 +773,7 @@ sub _targets_update {

# Updates a target.
sub _target_update {
my ($attrib_ref, $target) = @_;
my ($attrib_ref, $checksum_method, $target) = @_;
my $file_type_util = $attrib_ref->{file_type_util_of}{$target->get_type()};
eval {$file_type_util->task_of()->{$target->get_task()}->main($target)};
if ($@) {
Expand All @@ -781,7 +786,7 @@ sub _target_update {
return $E->throw($E->BUILD_TARGET, $target);
}
$target->set_status($target->ST_MODIFIED);
my $checksum = $UTIL->file_md5($target->get_path());
my $checksum = $UTIL->file_checksum($target->get_path(), $checksum_method);
if ($target->get_checksum() && $checksum eq $target->get_checksum()) {
$target->set_status($target->ST_UNCHANGED);
if ($target->get_path_of_prev()) {
Expand All @@ -804,6 +809,7 @@ sub _targets_manager_funcs {
my ($stack_ref, $state_hash_ref)
= _targets_select($attrib_ref, $m_ctx, $ctx, \@targets);

my $checksum_method = _prop($attrib_ref, 'checksum-method', $ctx);
my $get_action_ref = sub {
STATE:
while (my $state = pop(@{$stack_ref})) {
Expand All @@ -818,7 +824,7 @@ sub _targets_manager_funcs {
$stat_hash_ref, $ctx, $target, $state_hash_ref, $stack_ref,
);
}
elsif (_target_check_ood($state, $state_hash_ref)) {
elsif (_target_check_ood($state, $state_hash_ref, $checksum_method)) {
_target_prep($state, $ctx);
$state->set_value($STATE->PENDING);
# Adds tasks that can be triggered by this task
Expand Down Expand Up @@ -1388,7 +1394,7 @@ sub _target_check_failed_dep {

# Returns true if $target is out of date.
sub _target_check_ood {
my ($state, $state_hash_ref) = @_;
my ($state, $state_hash_ref, $checksum_method) = @_;
my $target = $state->get_target();
# Dependencies
my $rc;
Expand Down Expand Up @@ -1423,7 +1429,7 @@ sub _target_check_ood {
my $prop_of_prev_hash_ref = $target->get_prop_of_prev_of();
( !$path_of_prev
|| !-e $path_of_prev
|| $UTIL->file_md5($path_of_prev) ne $checksum
|| $UTIL->file_checksum($path_of_prev, $checksum_method) ne $checksum
|| $UTIL->hash_cmp($prop_hash_ref, $prop_of_prev_hash_ref)
);
}
Expand Down
32 changes: 24 additions & 8 deletions lib/FCM/Util.pm
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ package FCM::Util;
use base qw{FCM::Class::CODE};

use Digest::MD5;
use Digest::SHA;
use FCM::Context::Event;
use FCM::Context::Locator;
use FCM::Util::ConfigReader;
Expand Down Expand Up @@ -51,6 +52,7 @@ our %ACTION_OF = (
config_reader => _util_of_func('config_reader', 'main'),
external_cfg_get => \&_external_cfg_get,
event => \&_event,
file_checksum => \&_file_checksum,
file_ext => \&_file_ext,
file_head => \&_file_head,
file_load => \&_file_load,
Expand Down Expand Up @@ -244,6 +246,20 @@ sub _event {
}
}

# Returns the checksum of the content in a file system path.
sub _file_checksum {
my ($attrib_ref, $path, $algorithm) = @_;
my $handle = _file_load_handle($attrib_ref, $path);
binmode($handle);
$algorithm ||= 'md5';
my $digest = $algorithm eq 'md5'
? Digest::MD5->new() : Digest::SHA->new($algorithm);
$digest->addfile($handle);
my $checksum = $digest->hexdigest();
close($handle);
return $checksum;
}

# Returns the file extension of a file system path.
sub _file_ext {
my ($attrib_ref, $path) = @_;
Expand Down Expand Up @@ -287,13 +303,7 @@ sub _file_load_handle {
# Returns the MD5 checksum of the content in a file system path.
sub _file_md5 {
my ($attrib_ref, $path) = @_;
my $handle = _file_load_handle($attrib_ref, $path);
binmode($handle);
my $digest = Digest::MD5->new();
$digest->addfile($handle);
my $checksum = $digest->hexdigest();
close($handle);
return $checksum;
_file_checksum($attrib_ref, $path, 'md5');
}

# Saves content to a file system path.
Expand Down Expand Up @@ -669,6 +679,12 @@ L<FCM::Context::Event|FCM::Context::Event> or a valid event code. If the former
is true, @args is not used, otherwise, @args should be the event arguments for
the specified event code.
=item $u->file_checksum($path, $algorithm)
Returns the checksum of $path. If $algorithm is not specified, the default
algorithm to use is MD5. Otherwise, any algorithm supported by Perl's
Digest::SHA module can be used.
=item $u->file_ext($path)
Returns file extension of $path. E.g.:
Expand All @@ -695,7 +711,7 @@ Returns a file handle for loading contents from $path.
=item $u->file_md5($path)
Returns the MD5 checksum of $path.
Deprecated. Equivalent to $u->file_checksum($path, 'md5').
=item $u->file_save($path, $content)
Expand Down
65 changes: 65 additions & 0 deletions t/fcm-make/48-build-sha1.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
#!/bin/bash
#-------------------------------------------------------------------------------
# (C) British Crown Copyright 2006-15 Met Office.
#
# This file is part of FCM, tools for managing and building source code.
#
# FCM is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# FCM is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with FCM. If not, see <http://www.gnu.org/licenses/>.
#-------------------------------------------------------------------------------
# Test "fcm make", build, using SHA1 checksum to determine changes.
#-------------------------------------------------------------------------------
. "$(dirname "$0")/test_header"

get-hello-checksum() {
PERL5LIB="${FCM_HOME}/lib" perl - <<'__PERL__'
use strict;
use warnings;
use IO::Uncompress::Gunzip qw{gunzip};
use Storable;
gunzip('.fcm-make/ctx.gz', 'ctx');
my $m_ctx = retrieve('ctx');
printf(
"%s %s\n",
$m_ctx->{'ctx_of'}{'build'}{'source_of'}{'hello.f90'}{'checksum'},
'src/hello.f90',
);
unlink('ctx')
__PERL__
}
#-------------------------------------------------------------------------------
tests 8
cp -r "${TEST_SOURCE_DIR}/${TEST_KEY_BASE}/"* .
#-------------------------------------------------------------------------------
TEST_KEY="${TEST_KEY_BASE}"
run_pass "${TEST_KEY}" fcm make
run_pass "${TEST_KEY}-sha1sum-check" sha1sum -c - <<<"$(get-hello-checksum)"
HELLO_O_MTIME="$(stat -c '%Y' 'build/o/hello.o')"
#-------------------------------------------------------------------------------
TEST_KEY="${TEST_KEY_BASE}-incr"
run_pass "${TEST_KEY}" fcm make
run_pass "${TEST_KEY}-sha1sum-check" sha1sum -c - <<<"$(get-hello-checksum)"
HELLO_O_MTIME_INCR="$(stat -c '%Y' 'build/o/hello.o')"
run_pass "${TEST_KEY}-mtime-of-hello.o" \
test "${HELLO_O_MTIME_INCR}" -eq "${HELLO_O_MTIME}"
#-------------------------------------------------------------------------------
TEST_KEY="${TEST_KEY_BASE}-incr-2"
sed -i 's/Hello/Greet/' 'src/hello.f90'
sleep 1 # In case computer is very fast, when everything can happen within 1s.
run_pass "${TEST_KEY}" fcm make
run_pass "${TEST_KEY}-sha1sum-check" sha1sum -c - <<<"$(get-hello-checksum)"
HELLO_O_MTIME_INCR="$(stat -c '%Y' 'build/o/hello.o')"
run_pass "${TEST_KEY}-mtime-of-hello.o" \
test "${HELLO_O_MTIME_INCR}" -gt "${HELLO_O_MTIME}"
#-------------------------------------------------------------------------------
exit 0
5 changes: 5 additions & 0 deletions t/fcm-make/48-build-sha1/fcm-make.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
steps=build
build.source=$HERE/src
build.prop{checksum-method} = sha1
build.prop{file-ext.bin}=
build.target{task} = link
3 changes: 3 additions & 0 deletions t/fcm-make/48-build-sha1/src/hello.f90
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
program hello
write(*, '(a)') 'Hello World!'
end program hello

0 comments on commit 47bfbb3

Please sign in to comment.