forked from samtools/htslib
-
Notifications
You must be signed in to change notification settings - Fork 0
/
NEWS
1592 lines (1134 loc) · 66.4 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Noteworthy changes in release a.b
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Noteworthy changes in release 1.14 (22nd October 2021)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Features and Updates
--------------------
* Added a keep option to bgzip to leave the original file untouched. This
brings bgzip into line with gzip. (PR #1331, thanks to Alex Petty)
* "endpos" has been added to the filter language, giving the position
of the rightmost mapped base as measured by the CIGAR string. For
unmapped reads it is the same as "pos". (PR #1307, thanks to John Marshall)
* Interfaces have been added to interpret the new base modification tags
added to the SAMtags document in samtools/hts-specs#418. (PR #1132)
* New API functions hts_flush()/sam_flush()/bcf_flush() for flushing output
htsFile/samFile/vcfFile streams. (PR #1326, thanks to John Marshall)
* The synced_bcf_reader now sorts lines with symbolic alleles by END tag as
well as POS. (PR #1321)
* Added synced_bcf_reader options BCF_SR_REGIONS_OVERLAP and
BCF_SR_TARGETS_OVERLAP for better control of records that start outside
the desired region but overlap it are handled. Fixes samtools/bcftools#1420
and samtools/bcftools#1421 raised by John Marshall. (PR #1327)
* HTSlib will now accept long-cigar CG:B: tags made by htsjdk which don't
quite follow the specification properly (using signed values instead of
unsigned). Thanks to Colin Diesh for reporting an example file. (PR #1317)
* The warning printed when the BGZF reader finds a file with no EOF block
has been changed to be less alarming. Unfortunately some third-party
BGZF encoders don't write EOF blocks at the end of files. Thanks to
Keiran Raine for reporting an example file. (PR #1323)
* The FASTA and FASTQ readers get an option to skip over the first item on
the header line, and use the second as the read name. It allows the original
name to be restored on some of the fastq files served from the European
Nucleotide Archive (ENA). (PR #1325)
* HTSlib is now more strict when parsing the VCF samples line (beginning
#CHROM). It will only accept tabs between the mandatory field names and
sample names must be separated with tabs. (PR #1328)
* HTSlib will now warn if it looks like the header has been corrupted
by diagnostic messages from the program that made it. This can happen when
using `nohup`, which by default mixes stdout and stderr into the same
stream. (PR#1339, thanks to John Marshall)
* File format detection will now recognise signatures for XZ, Zstd and D4
files (note that HTSlib will not read them yet). (PR #1340, thanks to
John Marshall)
Build changes
-------------
These are compiler, configuration and makefile based changes.
* Some redundant tests have been removed from the test harness, speeding it up.
(PR #1308)
* The version.sh script now works better on shallow checkouts. (PR #1324)
* A check-untracked Makefile target has been added to catch untracked files
(mostly) left by the test harness. (PR #1324)
Bug fixes
---------
* Fixed a case where flushing the thread pool could very occasionally cause
a deadlock. (PR #1309)
* Fixed a bug where some CRAM files could fail to decode if the required_fields
option was in use. Thanks to Matt Sexton for reporting the issue.
(PR #1314, fixes samtools/samtools#1475)
* Fixed a regression where the S3 plugin could not read public files unless
you supplied some Amazon credentials. Thanks to Chris Saunders for reporting.
(PR #1332, fixes samtools/samtools#1491)
* Fixed a possible CRAM thread deadlock discovered by @ryancaicse.
(PR #1330, fixes #1329)
* Some set-but-unused variables have been removed. (PR #1334)
* Fixed a bug which prevented "flag.read2" from working in the filter
language unless it was at the end of the expression. Thanks to Vamsi Kodali
for reporting the issue. (PR #1342)
* Fixed a memory leak that could happen if CRAM fails to inflate a LZMA
block. (PR #1340, thanks to John Marshall)
Noteworthy changes in release 1.13 (7th July 2021)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Features and Updates
--------------------
* In case a PG header line has multiple ID tags supplied by other applications,
the header API now selects the first one encountered as the identifying tag
and issues a warning when detecting subsequent ID tags.
(#1256; fixed samtools/samtools#1393)
* VCF header reading function (vcf_hdr_read) no longer tries to download a
remote index file by default.
(#1266; fixes #380)
* Support reading and writing FASTQ format in the same way as SAM, BAM or CRAM.
Records read from a FASTQ file will be treated as unmapped data.
(#1156)
* Added GCP requester pays bucket access. Thanks to @indraniel.
(#1255)
* Made mpileup's overlap removal choose which copy to remove at random instead
of always removing the second one. This avoids strand bias in experiments
where the +ve and -ve strand reads always appear in the same order.
(#1273; fixes samtools/bcftools#1459)
* It is now possible to use platform specific BAQ parameters. This also
selects long-read parameters for read lengths bigger than 1kb, which helps
bcftools mpileup call SNPs on PacBio CCS reads.
(#1275)
* Improved bcf_remove_allele_set. This fixes a bug that stopped iteration over
alleles prematurely, marks removed alleles as 'missing' and does automatic
lazy unpacking.
(#1288; fixes #1259)
* Improved compression metrics for unsorted CRAM files. This improves the
choice of codecs when handling unsorted data.
(#1291)
* Linear index entries for empty intervals are now initialised with the file
offset in the next non-empty interval instead of the previous one. This
may reduce the amount of data iterators have to discard before reaching
the desired region, when the starting location is in a sequence gap.
Thanks to @carsonh for reporting the issue.
(#1286; fixes #486)
* A new hts_bin_level API function has been added, to compute the level of a
given bin in the binning index.
(#1286)
* Related to the above, a new API method, hts_idx_nseq, now returns the total
number of contigs from an index.
(#1295 and #1299)
* Added bracket handling to bcf_hdr_parse_line, for use with ##META lines.
Thanks to Alberto Casas Ortiz.
(#1240)
Build changes
-------------
These are compiler, configuration and makefile based changes.
* HTSlib now uses libhtscodecs release 1.1.1.
* Added a curl/curl.h check to configure and improved INSTALL documentation on
build options. Thanks to Melanie Kirsche and John Marshall.
(#1265; fixes #1261)
* Some fixes to address GCC 11.1 warnings.
(#1280, #1284, #1285; fixes #1283)
* Supports building HTSlib in a separate directory. Thanks to John Marshall.
(#1277; fixes #231)
* Supports building HTSlib on MinGW 32-bit environments. Thanks to
John Marshall.
(#1301)
Bug fixes
---------
* Fixed hts_itr_query() et al region queries: fixed bug introduced in
HTSlib 1.12, which led to iterators producing very few reads for some
queries (especially for larger target regions) when unmapped reads were
present. HTSlib 1.11 had a related problem in which iterators would omit
a few unmapped reads that should have been produced; cf #1142.
Thanks to Daniel Cooke for reporting the issue.
(#1281; fixes #1279)
* Removed compressBound assertions on opening bgzf files. Thanks to
Gurt Hulselmans for reporting the issue.
(#1258; fixed #1257)
* Duplicate sample name error message for a VCF file now only displays the
duplicated name rather the entire same name list.
(#1262; fixes samtools/bcftools#1451)
* Fix to make samtools cat work on CRAMs again.
(#1276; fixes samtools/samtools#1420)
* Fix for a double memory free in SAM header creation. Thanks to @ihsineme.
(#1274)
* Prevent assert in bcf_sr_set_regions. Thanks to Dr K D Murray.
(#1270)
* Fixed crash in knet_open() etc stubs. Thanks to John Marshall.
(#1289)
* Fixed filter expression "cigar" on unmapped reads. Stop treating an empty
CIGAR string as an error. Thanks to Chang Y for reporting the issue.
(#1298, fixes samtools/samtools#1445)
* Bug fixes in the bundled copy of htscodecs:
- Fixed an uninitialized access in the name tokeniser decoder.
(samtools/htscodecs#23)
- Fixed a bug with name tokeniser and variable number of names per slice,
causing it to incorrectly report an error on certain valid inputs.
(samtools/htscodecs#24)
Noteworthy changes in release 1.12 (17th March 2021)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Features and Updates
--------------------
* Added experimental CRAM 3.1 and 4.0 support. (#929)
These should not be used for long term data storage as the
specification still needs to be ratified by GA4GH and may be subject
to changes in format. (This is highly likely for 4.0). However it
may be tested using:
test/test_view -t ref.fa -C -o version=3.1 in.bam -p out31.cram
For smaller but slower files, try varying the compression profile
with an additional "-o small". Profile choices are fast, normal,
small and archive, and can be applied to all CRAM versions.
* Added a general filtering syntax for alignment records in SAM/BAM/CRAM
readers. (#1181, #1203)
An example to find chromosome spanning read-pairs with high mapping
quality: 'mqual >= 30 && mrname != rname'
To find significant sized deletions:
'cigar =~ "[0-9]{2}D"' or 'rlen - qlen > 10'.
To report duplicates that aren't part of a "proper pair":
'flag.dup && !flag.proper_pair'
More details are in the samtools.1 man page under "FILTER EXPRESSIONS".
* The knet networking code has been removed. It only supported the http
and ftp protocols, and a better and safer alternative using libcurl
has been available since release 1.3. If you need access to ftp:// and
http:// URLs, HTSlib should be built with libcurl support. (#1200)
* The old htslib/knetfile.h interfaces have been marked as deprecated. Any
code still using them should be updated to use hFILE instead. (#1200)
* Added an introspection API for checking some of the capabilities provided
by HTSlib. (#1170) Thanks also to John Marshall for contributions. (#1222)
- `hfile_list_schemes`: returns the number of schemes found
- `hfile_list_plugins`: returns the number of plugins found
- `hfile_has_plugin`: checks if a specific plugin is available
- `hts_features`: returns a bit mask with all available features
- `hts_test_feature`: test if a feature is available
- `hts_feature_string`: return a string summary of enabled features
* Made performance improvements to `probaln_glocal` method, which
speeds up mpileup BAQ calculations. (#1188)
- Caching of reused loop variables and removal of loop invariants
- Code reordering to remove instruction latency.
- Other refactoring and tidyups.
* Added a public method for constructing a BAM record from the
component pieces. Thanks to Anders Kaplan. (#1159, #1164)
* Added two public methods, `sam_parse_cigar` and `bam_parse_cigar`, as part of
a small CIGAR API (#1169, #1182). Thanks to Daniel Cameron for input. (#1147)
* HTSlib, and the included htsfile program, will now recognise the old
RAZF compressed file format. Note that while the format is detected,
HTSlib is unable to read it. It is recommended that RAZF files are
uncompressed with `gunzip` before using them with HTSlib. Thanks to
John Marshall (#1244); and Matthew J. Oldach who reported problems
with uncompressing some RAZF files (samtools/samtools#1387).
* The S3 plugin now has options to force the address style. It will recognise
the addressing_style and host_bucket entries in the respective aws
.credentials and s3cmd .s3cfg files. There is also a new HTS_S3_ADDRESS_STYLE
environment variable. Details are in the htslib-s3-plugin.7 man file (#1249).
Build changes
-------------
These are compiler, configuration and makefile based changes.
* Added new Makefile targets for the applications that embed HTSlib and
want to run its test suite or clean its generated artefacts. (#1230, #1238)
* The CRAM codecs are now obtained via the htscodecs submodule, hence
when cloning it is now best to use "git clone --recursive". In an
existing clone, you may use "git submodule update --init" to obtain
the htscodecs submodule checkout.
* Updated CI test configuration to recurse HTSlib submodules. (#1359)
* Added Cirrus-CI integration as a replacement for Travis, which was
phased out. (#1175; #1212)
* Updated the Windows image used by Appveyor to 'Visual Studio 2019'. (#1172;
fixed #1166)
* Fixed a buglet in configure.ac, exposed by the release 2.70 of autoconf.
Thanks to John Marshall. (#1198)
* Fixed plugin linking on macOS, to prevent symbol conflict when linking
with a static HTSlib. Thanks to John Marshall. (#1184)
* Fixed a clang++9 error in `cram_io.h`. Thanks to Pjotr Prins. (#1190)
* Introduced $(ALL_CPPFLAGS) to allow for more flexibility in setting the
compiler flags. Thanks to John Marshall. (#1187)
* Added 'fall through' comments to prevent warnings issued by Clang on
intentional fall through case statements, when building with
`-Wextra flag`. Thanks to John Marshall. (#1163)
* Non-configure builds now define _XOPEN_SOURCE=600 to allow them to work
when the `gcc -std=c99` option is used. Thanks to John Marshall. (#1246)
Bug fixes
---------
* Fixed VCF `#CHROM` header parsing to only separate columns at tab characters.
Thanks to Sam Morris for reporting the issue.
(#1237; fixed samtools/bcftools#1408)
* Fixed a crash reported in `bcf_sr_sort_set`, which expects REF to be present.
(#1204; fixed samtools/bcftools#1361)
* Fixed a bcf synced reader bug when filtering with a region list, and
the first record for a chromosome had the same position as the last
record for the previous chromosome. (#1254; fixed samtools/bcftools#1441)
* Fixed a bug in the overlapping logic of mpileup, dealing with iterating over
CIGAR segments. Thanks to `@wulj2` for the analysis. (#1202; fixed #1196)
* Fixed a tabix bug that prevented setting the correct number of lines to be
skipped in a region file. Thanks to Jim Robinson for reporting it. (#1189;
fixed #1186)
* Made `bam_itr_next` an alias for `sam_itr_next`, to prevent it from crashing
when working with htsFile pointers. Thanks to Torbjörn Klatt for
reporting it. (#1180; fixed #1179)
* Fixed once per outgoing multi-threaded block `bgzf_idx_flush` assertion, to
accommodate situations when a single record could span multiple blocks.
Thanks to `@lacek`. (#1168; fixed samtools/samtools#1328)
* Fixed assumption of pthread_t being a non-structure, as permitted by POSIX.
Thanks also to John Marshall and Anders Kaplan. (#1167, #1153, #1153)
* Fixed the minimum offset of a BAI index bin, to account for unmapped reads.
Thanks to John Marshall for spotting the issue. (#1158; fixed #1142)
* Fixed the CRLF handling in `sam_parse_worker` method. Thanks to
Anders Kaplan. (#1149; fixed #1148)
* Included unistd.h and errno.h directly in HTSlib files, as opposed to
including them indirectly, via third party code. Thanks to
Andrew Patterson (#1143) and John Marshall (#1145).
Noteworthy changes in release 1.11 (22nd September 2020)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Features and Updates
--------------------
* Support added for remote reference files. fai_path() can take a remote
reference file and will return the corresponding index file. Remote indexes
can be handled by refs_load_fai(). UR tags in @SQ lines can now be set to
remote URIs. (#1017)
* Added tabix --separate-regions option, which adds header comment lines
separating different regions' output records when multiple target regions
are supplied on the command line. (#1108)
* Added tabix --cache option to set a BGZF block cache size. Most beneficial
when the -R option is used and the same blocks need to be re-read multiple
times. (#1053)
* Improved error checking in tabix and added a --verbosity option so
it is possible to change the amount of logging when it runs. (#1040)
* A note about the maximum chromosome length usable with TBI indexes has been
added to the tabix manual page. Thanks to John Marshall. (#1070)
* New method vcf_open_mode() changes the opening mode of a variant file
based on its file extension. Similar to sam_open_mode(). (#1096)
* The VCF parser has been made faster and easier to maintain. (#1057)
* bcf_record_check() has been made faster, giving a 15% speed increase when
reading an uncompressed BCF file. (#1130)
* The VCF parser now recognises the "<NON_REF>" symbolic allele produced
by GATK. (#1045)
* Support has been added for simultaneous reading of unindexed VCF/BCF files
when using the synced_bcf_reader interface. Input files must have the
chromosomes in the same order as each other and be consistent with the order
of sequences in the header. (#1089)
* The VCF and BCF readers will now attempt to fix up invalid INFO/END tags
where the stored END value is less than POS, resulting in an apparently
negative record length. Such files have been generated by programs which
used END incorrectly, and by broken lift-over processes that failed to
update any END tags present. (#1021; fixed samtools/bcftools#1154)
* The htsFile interface can now detect the crypt4gh encrypted format (see
https://samtools.github.io/hts-specs/crypt4gh.pdf). If HTSlib is
built with external plug-in support, and the hfile_crypt4gh plug-in is
present, the file will be passed to it for decryption. The plug-in
can be obtained from https://github.com/samtools/htslib-crypt4gh. (#1046)
* hts_srand48() now seeds the same POSIX-standard sequences of pseudo-random
numbers regardless of platform, including on OpenBSD where plain srand48()
produces a different cryptographically-strong non-deterministic sequence.
Thanks to John Marshall. (#1002)
* Iterators now work with 64 bit positions. (#1018)
* Improved the speed of range queries when using BAI indexes by
making better use of the linear index data included in the file.
The best improvement is on low-coverage data. (#1031)
* Alignments which consume no reference bases are now considered to have
length 1. This would make such alignments cover 1 reference position in
the same manner as alignments that are unmapped or have no CIGAR strings.
These alignments can now be returned by iterator-based queries. Thanks
to John Marshall. (#1063; fixed samtools/samtools#1240, see also
samtools/hts-specs#521).
* A bam_set_seqi() function to modify a single base in the BAM structure
has been added. This is a companion function to bam_seqi(). (#1022)
* Writing SAM format is around 30% faster. (#1035)
* Added sam_format_aux1() which converts a BAM aux tag to a SAM format string.
(#1134)
* bam_aux_update_str() no longer requires NUL-terminated strings. It
is also now possible to create tags containing part of a longer string.
(#1088)
* It is now possible to use external plug-ins in language bindings that
dynamically load HTSlib. Note that a side-effect of this change is that
some plug-ins now link against libhts.so, which means that they have to be
able to find the shared library when they are started up. Thanks to
John Marshall. (#1072)
* bgzf_close(), and therefore hts_close(), will now return non-zero when
closing a BGZF handle on which errors have been detected. (Part of #1117)
* Added a special case to the kt_fisher_exact() test for when the table
probability is too small to be represented in a double. This fixes a
bug where it would, for some inputs, fail to correctly determine which
side of the distribution the table was on resulting in swapped p-values
being returned for the left- and right-tailed tests. The two-tailed
test value was not affected by this problem. (#1126)
* Improved error diagnostics in the CRAM decoder (#1042), BGZF (#1049),
the VCF and BCF readers (#1059), and the SAM parser (#1073).
* ks_resize() now allocates 1.5 times the requested size when it needs
to expand a kstring instead of rounding up to the next power of two.
This has been done mainly to make the inlined function smaller, but it
also reduces the overhead of storing data in kstrings at the expense of
possibly needing a few more reallocations. (#1129)
CRAM improvements
-----------------
* Delay CRAM crc32 checks until the data actually needs to be used. With
other changes this leads to a 20x speed up in indexing and other sub-query
based actions. (#988)
* CRAM now handles the transition from mapped to unmapped data in a better
way, improving compression of the unmapped data. (#961)
* CRAM can now use libdeflate. (#961)
* Fixed bug in MD tag generation with "b" read feature codes, causing the
numbers in the tag to be too large. Note that HTSlib never uses this
feature code so it is unlikely that this bug would be seen on real data.
The problem was found when testing against hand-crafted CRAM files. (#1086)
* Fixed a regression where the CRAM multi-region iterator became much less
efficient when using threads. It now works more like the single iterator
and does not preemptively decode the next container unless it will be used.
(#1061)
* Set CRAM default quality in lossy quality modes. If lossy quality is enabled
and 'B', 'q' or 'Q' features are used, CRAM starts off with QUAL being all 255
(as per BAM spec and "*" quality) and then modifies individual qualities as
dictated by the specific features.
However that then produces ASCII quality " " (space, q=-1) for the unmodified
bases. Instead ASCII quality "?" (q=30) is used, as per HTSJDK. Quality 255
is still used for sequences with no modifications at all. (#1094)
Build changes
-------------
These are compiler, configuration and makefile based changes.
* `make all` now also builds htslib_static.mk and htslib-uninstalled.pc.
Thanks to John Marshall. (#1011)
* Various cppcheck-1.90 warnings have been fixed. (#995, #1011)
* HTSlib now prefers its own headers when being compiled, fixing build
failures on machines that already had a system-installed HTSlib. Thanks to
John Marshall. (#1078; fixed #347)
* Define HTSLIB_EXPORT without using a helper macro to reduce the length of
compiler diagnostics that mention exported functions. Thanks to
John Marshall. (#1029)
* Fix dirty default build by including latest pkg.m4 instead of using
aclocal.m4. Thanks to Damien Zammit. (#1091)
* Struct tags have been added to htslib/*.h public typedefs. This makes it
possible to forward declare htsFile without including htslib/hts.h. Thanks
to Lucas Czech and John Marshall. (#1115; fixed #1106)
* Fixed compiler warnings emitted by the latest gcc and clang releases
when compiling HTSlib, along with some -Wextra warnings in the public
include files. Thanks to John Marshall. (#1066, #1063, #1083)
Bug fixes
---------
* Fixed hfile_libcurl breakage when using libcurl 7.69.1 or later. Thanks to
John Marshall for tracking down the exact libcurl change that caused the
incompatibility. (#1105; fixed samtools/samtools#1254 and
samtools/samtools#1284)
* Fixed overflows kroundup32() and kroundup_size_t() which caused them to
return zero when rounding up values where the most significant bit was
set. When this happens they now return the highest value that can
be stored (#1044). All of the kroundup macro definitions have also been
gathered together into a unified implementation (#1051).
* Fixed missing return parameter value in idx_test_and_fetch(). Thanks to
Lilian Janin. (#1014)
* Fixed crashes due to inconsistent selection between BGZF and plain (hFILE)
interfaces when reading files. [fuzz] (#1019)
* Added and/or fixed byte swapping code for big-endian platforms. Thanks
to Jun Aruga, John Marshall, Michael R Crusoe and Gianfranco Costamagna
for their help. (#1023; fixed #119 and #355)
* Fixed a problem with multi-threaded on-the-fly indexes which would
occasionally write virtual offsets pointing at the end of a BGZF block.
Attempting to read from such an offset caused EOF to be incorrectly
reported. These offsets are now handled correctly, and the indexer
has been updated to avoid generating them. (#1028; fixed
samtools/samtools#1197)
* In sam_hdr_create(), free newly allocated SN strings when encountering an
error. [fuzz] (#1034)
* Prevent double free in case of idx_test_and_fetch() failure. Thanks to
@fanwayne for the bug report. (#1047; fixed #1033)
* In the header, link a new PG line only to valid chains. Prevents an
explosive growth of PG lines on headers where PG lines are already present
but not linked together correctly. (#1062; fixed samtools/samtools#1235)
* Also in the header, when calling sam_hdr_update_line(), update target arrays
only when the name or length is changed. (#1007)
* Fixed buffer overflows in CRAM MD5 calculation triggered by
files with invalid compression headers, or files with embedded
references that were one byte too short. [fuzz] (#1024, #1068)
* Fix mpileup regression between 1.9 and 1.10 where overlap detection
was incorrectly skipped on reads where RNEXT, PNEXT and TLEN were
set to the "unavailable" values ("*", 0, 0 in SAM). (#1097)
* kputs() now checks for null pointer in source string. [fuzz] (#1087)
* Fix potential bcf_update_alleles() crash on 0 alleles. Thanks to
John Marshall. (#994)
* Added bcf_unpack() calls to some bcf_update functions to fix a bug
where updates made after a call to bcf_dup() could be lost. (#1032;
fixed #1030)
* Error message typo "Number=R" instead of "Number=G" fixed in
bcf_remove_allele_set(). Thanks to Ilya Vorontsov. (#1100)
* Fixed crashes that could occur in BCF files that use IDX= header annotations
to create a sparse set of CHROM, FILTER or FORMAT indexes, and
include records that use one of the missing index values. [fuzz] (#1092)
* Fixed potential integer overflows in the VCF parser and ensured that
the total length of FORMAT fields cannot go over 2Gbytes. [fuzz] (#1044,
#1104; latter is CVE-2020-36403 affecting HTSlib versions 1.10 to 1.10.2)
* Download index files atomically in idx_test_and_fetch(). This prevents
corruption when running parallel jobs on S3 files. Thanks to John Marshall.
(#1112; samtools/samtools#1242).
* The pileup constructor callback is now given the copy of the bam1_t struct
made by pileup instead of the original one passed to bam_plp_push(). This
makes it the same as the one passed to the destructor and ensures that
cached data, for example the location of an aux tag, will remain valid.
(#1127)
* Fixed possible error in code_sort() on negative CRAM Huffman code
length. (#1008)
* Fixed possible undefined shift in cram_byte_array_stop_decode_init(). (#1009)
* Fixed a bug where range queries to the end of a given reference
would return incorrect results on CRAM files. (#1016;
fixed samtools/samtools#1173)
* Fixed an integer overflow in cram_read_slice(). [fuzz] (#1026)
* Fixed a memory leak on failure in cram_decode_slice(). [fuzz] (#1054)
* Fixed a regression which caused cram_transcode_rg() to fail, resulting
in a crash in "samtools cat" on CRAM files. (#1093;
fixed samtools/samtools#1276)
* Fixed an undersized string reallocation in the threaded SAM reader which
caused it to crash when reading SAM files with very long lines. Numerous
memory allocation checks have also been added. (#1117)
Noteworthy changes in release 1.10.2 (19th December 2019)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is a release fix that corrects minor inconsistencies discovered in
previous deliverables.
Noteworthy changes in release 1.10.1 (17th December 2019)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The support for 64-bit coordinates in VCF brought problems for files
not conforming to VCF/BCF specification. While previous versions would
make out-of-range values silently overflow creating nonsense values
but parseable file, the version 1.10 would silently create an invalid BCF.
Noteworthy changes in release 1.10 (6th December 2019)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Brief summary
-------------
There are many changes in this release, so the executive summary is:
* Addition of support for references longer than 2Gb (NB: SAM and VCF
formats only, not their binary counterparts). This may need changes
in code using HTSlib. See README.large_positions.md for more information.
* Added a SAM header API.
* Major speed up to SAM reading and writing. This also now supports
multi-threading.
* We can now auto-index on-the-fly while writing a file. This also
includes to bgzipped SAM.gz.
* Overhaul of the S3 interface, which now supports version 4
signatures. This also makes writing to S3 work.
These also required some ABI changes. See below for full details.
Features / updates
------------------
* A new SAM/BAM/CRAM header API has been added to HTSlib, allowing header
data to be updated without having to parse or rewrite large parts of the
header text. See htslib/sam.h for function definitions and
documentation. (#812)
The header typedef and several pre-existing functions have been renamed
to have a sam_hdr_ prefix: sam_hdr_t, sam_hdr_init(), sam_hdr_destroy(),
and sam_hdr_dup(). (The existing bam_hdr_-prefixed names are still
provided for compatibility with existing code.) (#887, thanks to
John Marshall)
* Changes to hfile_s3, which provides support for the AWS S3 API. (#839)
- hfile_s3 now uses version 4 signatures by default. Attempting to write to
an S3 bucket will also now work correctly. It is possible to force
version 2 signatures by creating environment variable HTS_S3_V2 (the exact
value does not matter, it just has to exist). Note that writing depends
on features that need version 4 signatures, so forcing version 2 will
disable writes.
- hfile_s3 will automatically retry requests where the region endpoint
was not specified correctly, either by following the 301 redirect (when
using path-style requests) or reading the 400 response (when using
virtual-hosted style requests and version 4 signatures). The first
region to try can be set by using the AWS_DEFAULT_REGION environment
variable, by setting "region" in ".aws/credentials" or by setting
"bucket_location" in ".s3cfg".
- hfile_s3 now percent-escapes the path component of s3:// URLs. For
backwards-compatibility it will ignore any paths that have already
been escaped (detected by looking for '%' followed by two hexadecimal
digits.)
- New environment variables HTS_S3_V2, HTS_S3_HOST, HTS_S3_S3CFG
and HTS_S3_PART_SIZE to force version-2 signatures, control the
S3 server hostname, the configuration file and upload chunk
sizes respectively.
* Numerous SAM format improvements.
- Bgzipped SAM files can now be indexed and queried. The library now
recognises sam.gz as a format name to ease this usage. (#718, #916)
- The SAM reader and writer now supports multi-threading via the
thread-pool. (#916)
Note that the multi-threaded SAM reader does not currently support seek
operations. Trying to do this (for example with an iterator range request)
will result in the SAM readers dropping back to single-threaded mode.
- Major speed up of SAM decoding and encoding, by around 2x. (#722)
- SAM format can now handle 64-bit coordinates and references. This
has implications for the ABI too (see below). Note BAM and CRAM
currently cannot handle references longer than 2Gb, however given
the speed and threading improvements SAM.gz is a viable workaround. (#709)
* We can now automatically build indices on-the-fly while writing
SAM, BAM, CRAM, VCF and BCF files. (Note for SAM and VCF this only
works when bgzipped.) (#718)
* HTSlib now supports the @SQ-AN header field, which lists alternative names
for reference sequences. This means given "@SQ SN:1 AN:chr1", tools like
samtools can accept requests for "1" or "chr1" equivalently. (#931)
* Zero-length files are no longer considered to be valid SAM files
(with no header and no alignments). This has been changed so that pipelines
such as `somecmd | samtools ...` with `somecmd` aborting before outputting
anything will now propagate the error to the second command. (#721, thanks
to John Marshall; #261 reported by Adrian Tan)
* Added support for use of non-standard index names by pasting the
data filename and index filename with ##idx##. For example
"/path1/my_data.bam##idx##/path2/my_index.csi" will open bam file
"/path1/my_data.bam" and index file "/path2/my_index.csi". (#884)
This affects hts_idx_load() and hts_open() functions.
* Improved the region parsing code to handle colons in reference
names. Strings can be disambiguated by the use of braces, so for
example when reference sequences called "chr1" and "chr1:100-200"
are both present, the regions "{chr1}:100-200" and "{chr1:100-200}"
unambiguously indicate which reference is being used. (#708)
A new function hts_parse_region() has been added along with
specialisations for sam_parse_region() and fai_parse_region().
* CRAM encoding now has additional checks for MD/NM validity. If
they are incorrect, it stores the (incorrect copy) verbatim so
round-trips "work". (#792)
* Sped up decoding of CRAM by around 10% when the MD tag is being
generated. (#874)
* CRAM REF_PATH now supports %Ns (where N is a single digit)
expansion in http URLs, similar to how it already supported this
for directories. (#791)
* BGZF now permits indexing and seeking using virtual offsets in
completely uncompressed streams. (#904, thanks to Adam Novak)
* bgzip now asks for extra confirmation before decompressing files
that don't have a known compression extension (e.g. .gz). This avoids
`bgzip -d foo.bam.bai` producing a foo.bam file that is very much not
a BAM-formatted file. (#927, thanks to John Marshall)
* The htsfile utility can now copy files (including to/from URLs using
HTSlib's remote access facilities) with the --copy option, in
addition to its existing uses of identifying file formats and
displaying sequence or variant data. (#756, thanks to John Marshall)
* Added tabix --min-shift option. (#752, thanks to Garrett Stevens)
* Tabix now has an -D option to disable storing a local copy of a
remote index. (#870)
* Improved support for MSYS Windows compiler environment. (#966)
* External htslib plugins are now supported on Windows. (#966)
API additions and improvements
------------------------------
* New API functions bam_set_mempolicy() and bam_get_mempolicy() have
been added. These allow more control over the ownership of bam1_t
alignment record data; see documentation in htslib/sam.h for more
information. (#922)
* Added more HTS_RESULT_USED checks, this time for VCF I/O. (#805)
* khash can now hash kstrings. This makes it easier to hash
non-NUL-terminated strings. (#713)
* New haddextension() filename extension API function. (#788, thanks to
John Marshall)
* New hts_resize() macro, designed to replace uses of hts_expand()
and hts_expand0(). (#805)
* Added way of cleaning up unused jobs in the thread pool via the new
hts_tpool_dispatch3() function. (#830)
* New API functions hts_reglist_create() and sam_itr_regarray() are added
to create hts_reglist_t region lists from `chr:<from>-<to>` type region
specifiers. (#836)
* Ksort has been improved to facilitate library use. See KSORT_INIT2
(adds scope / namespace capabilities) and KSORT_INIT_STATIC interfaces.
(#851, thanks to John Marshall)
* New kstring functions (#879):
KS_INITIALIZE - Initializer for structure assignment
ks_initialize() - Initializer for pointed-to kstrings
ks_expand() - Increase kstring capacity by a given amount
ks_clear() - Set kstring length to zero
ks_free() - Free the underlying buffer
ks_c_str() - Returns the kstring buffer as a const char *,
or an empty string if the length is zero.
* New API functions hts_idx_load3(), sam_index_load3(), tbx_index_load3()
and bcf_index_load3() have been added. These allow control of whether
remote indexes should be cached locally, and allow the error message
printed when the index does not exist to be suppressed. (#870)
* Improved hts_detect_format() so it no longer assumes all text is
SAM unless positively identified otherwise. It also makes a stab
at detecting bzip2 format and identifying BED, FASTA and FASTQ
files. (#721, thanks to John Marshall; #200, #719 both reported by
Torsten Seemann)
* File format errors now set errno to EFTYPE (BSD, MacOS) when
available instead of ENOEXEC. (#721)
* New API function bam_set_qname (#942)
* In addition to the existing hts_version() function, which reflects the
HTSlib version being used at runtime, <htslib/hts.h> now also provides
HTS_VERSION, a preprocessor macro reflecting the HTSlib version that
a program is being compiled against. (#951, thanks to John Marshall; #794)
ABI changes
-----------
This release contains a number of things which change the Application
Binary Interface (ABI). This means code compiled against an earlier
library will require recompiling. The shared library soversion has
been bumped.
* On systems that support it, the default symbol visibility has been
changed to hidden and the only exported symbols are ones that form part
of the officially supported ABI. This is to make clear exactly which
symbols are considered parts of the library interface. It also
helps packagers who want to check compatibility between HTSlib versions.
(#946; see for example issues #311, #616, and #695)
* HTSlib now supports 64 bit reference positions. This means several
structures, function parameters, and return values have been made bigger
to allow larger values to be stored. While most code that uses
HTSlib interfaces should still build after this change, some alterations
may be needed - notably to printf() formats where the values of structure
members are being printed. (#709)
Due to file format limitations, large positions are only supported
when reading and writing SAM and VCF files.
See README.large_positions.md for more information.
* An extra field has been added to the kbitset_t struct so bitsets can
be made smaller (and later enlarged) without involving memory allocation.
(#710, thanks to John Marshall)
* A new field has been added to the bam_pileup1_t structure to keep track
of which CIGAR operator is being processed. This is used by a new
bam_plp_insertion() function which can be used to return the sequence of
any inserted bases at a given pileup location. If the alignment includes
CIGAR P operators, the returned sequence will include pads. (#699)
* The hts_itr_t and hts_itr_multi_t structures have been merged and can be
used interchangeably. Extra fields have been added to hts_itr_t to support
this. hts_itr_multi_t is now a typedef for hts_itr_t; sam_itr_multi_next()
is now an alias for sam_itr_next() and hts_itr_multi_destroy() is an alias
for hts_itr_destroy(). (#836)
* An improved regidx interface has been added. To allow this, struct
reg_t has been removed, regitr_t has been modified and various new
API functions have been added to htslib/regidx.h. While parts of
the old regidx API have been retained for backwards compatibility,
it is recommended that all code using regidx should be changed to use
the new interface. (#761)
* Elements in the hts_reglist_t structure have been reordered slightly
so that they pack together better. (#761)
* bgzf_utell() and bgzf_useek() now use type off_t instead of long for
the offset. This allows them to work correctly on files longer than
2G bytes on Windows and 32-bit Linux. (#868)
* A number of functions that used to return void now return int so that
they can report problems like memory allocation failures. Callers
should take care to check the return values from these functions. (#834)
The affected functions are:
ksort.h: ks_introsort(), ks_mergesort()
sam.h: bam_mplp_init_overlaps()
synced_bcf_reader.h: bcf_sr_regions_flush()
vcf.h: bcf_format_gt(), bcf_fmt_array(),
bcf_enc_int1(), bcf_enc_size(),
bcf_enc_vchar(), bcf_enc_vfloat(), bcf_enc_vint(),
bcf_hdr_set_version(), bcf_hrec_format()
vcfutils.h: bcf_remove_alleles()
* bcf_set_variant_type() now outputs VCF_OVERLAP for spanning
deletions (ALT=*). (#726)
* A new field (hrecs) has been added to the bam_hdr_t structure for
use by the new header API. The old sdict field is now not used and
marked as deprecated. The l_text field has been changed from uint32_t
to size_t, to allow for very large headers in SAM files. The text
and l_text fields have been left for backwards compatibility, but
should not be accessed directly in code that uses the new header API.
To access the header text, the new functions sam_hdr_length() and
sam_hdr_str() should be used instead. (#812)
* The old cigar_tab field is now marked as deprecated; use the new
bam_cigar_table[] instead. (#891, thanks to John Marshall)
* The bam1_core_t structure's l_qname and l_extranul fields have been
rearranged and enlarged; l_qname still includes the extra NULs.
(Almost all code should use bam_get_qname(), bam_get_cigar(), etc,
and has no need to use these fields directly.) HTSlib now supports
the SAM specification's full 254 QNAME length again. (#900, thanks
to John Marshall; #520)
* bcf_index_load() no longer tries the '.tbi' suffix when looking for
BCF index files (.tbi indexes are for text files, not binary BCF). (#870)
* htsFile has a new 'state' member to support SAM multi-threading. (#916)
* A new field has been added to the bam1_t structure, and others
have been rearranged to remove structure holes. (#709; #922)
Bug fixes
---------
* Several BGZF format fixes:
- Support for multi-member gzip files. (#744, thanks to Adam Novak; #742)
- Fixed error handling code for native gzip formatted files. (64c4927)
- CRCs checked when threading too (previously only when non-threaded). (#745)
- Made bgzf_useek function work with threads. (#818)
- Fixed rare threading deadlocks. (#831)
- Reading of very short files (<28 bytes) that do not contain an EOF block.
(#910)
* Fixed some thread pool deadlocks caused by race conditions. (#746, #906)