BS ISO 28500:2017

Foreword
Introduction
1 Scope
2 Normative references
3 Terms, definitions and abbreviated terms
4 File and record model
5 Named fields
6 WARC record types
7 Record segmentation
8 WARC file name, size and compression
Annex A (informative) - Use cases for writing WARC
        records
Annex B (informative) - Examples of WARC records
Annex C (informative) - WARC file size and name
        recommendations
Annex D (informative) - Compression recommendations
Bibliography

Defines the WARC file format: - to store both the payload content and control information from mainstream Internet application layer protocols, such as the HTTP, DNS, and FTP; - to store arbitrary metadata linked to other stored data (e.g. subject classifier, discovered language, encoding); - to support data compression and maintain data record integrity; - to store all control information from the harvesting protocol (e.g. request headers), not just response information; - to store the results of data transformations linked to other stored data; - to store a duplicate detection event linked to other stored data (to reduce storage in the presence of identical or substantially similar resources); - to be extended without disruption to existing functionality; - to support handling of overly long records by truncation or segmentation, where desired.

This document specifies the WARC file format:

• to store both the payload content and control information from mainstream Internet application layer protocols, such as the HTTP, DNS, and FTP;

• to store arbitrary metadata linked to other stored data (e.g. subject classifier, discovered language, encoding);

• to support data compression and maintain data record integrity;

• to store all control information from the harvesting protocol (e.g. request headers), not just response information;

• to store the results of data transformations linked to other stored data;

• to store a duplicate detection event linked to other stored data (to reduce storage in the presence of identical or substantially similar resources);

• to be extended without disruption to existing functionality;

• to support handling of overly long records by truncation or segmentation, where desired.