Principes de base pour la gestion des fichiers volumineux
La fragmentation implique de diviser un fichier important en segments de taille prédéfinie, facilitant ainsi le transfert. La reprise après interruption permet de reprenrde un transfert interrompu en vérifiant l'existence des segments déjà envoyés. Lors de la fusion, les segments sont recombinés dans l'ordre séquentiel, avec une gestion asynchrone pour assurer la complétude. L'optimisation par hachage MD5 permet d'éviter le transfert de fichiers déjà présents sur le serveur.
Initialisation du projet Spring Boot
Créez un nouveau projet Spring Boot via Spring Initializr, en sélectionnant les dépendances web et test. Configurez l'application avec les propriétés appropriées pour gérer les uploads volumineux.
Configuraton des dépendances et paramètres
Ajoutez les dépendances nécessaires dans le fichier pom.xml pour la gestion des fichiers et les requêtes HTTP.
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>commons-fileupload</groupId>
<artifactId>commons-fileupload</artifactId>
<version>1.4</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.11.0</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents.client5</groupId>
<artifactId>httpclient5</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>
Modifiez le fichier application.yml pour ajuster les limites de taille des fichiers et désactiver la configuration par défaut de Spring.
server:
port: 8020
spring:
servlet:
multipart:
enabled: false
max-file-size: 2GB
max-request-size: 20GB
Il est crucial de définir max-file-size et max-request-size pour éviter les erreurs avec les fichiers importants.
Interface utilisateur pour l'upload
Intégrez une bibliothèque comme WebUploader côté client pour gérer la fragmentation et l'envoi concurrent des segments.
<html lang="fr">
<head>
<meta charset="UTF-8">
<title>Upload et téléchargement de fichiers</title>
<link rel="stylesheet" href="webuploader.css">
<script src="jquery.min.js"></script>
<script src="webuploader.min.js"></script>
<style>
#drop-zone { width: 120px; height: 60px; background: #a8d8ea; cursor: pointer; }
</style>
</head>
<body>
<div id="drop-zone">Glissez et déposez des fichiers ici</div>
<button id="select-btn">Sélectionner des fichiers</button>
<div id="file-list"></div>
<hr>
<a href="/data/simple-download">Téléchargement simple</a>
<a href="/data/chunked-download" target="_blank">Téléchargement fragmenté</a>
<script>
const uploader = WebUploader.create({
auto: true,
server: '/data/upload-chunk',
pick: '#select-btn',
dnd: '#drop-zone',
chunked: true,
threads: 4,
chunkRetry: 3,
fileVal: 'fileData'
});
uploader.on('uploadProgress', function(file, percent) {
$('#file-list').append('<p>Progression: ' + Math.round(percent * 100) + '%</p>');
});
</script>
</body>
</html>
Contrôleur pour les opérations de fichiers
Définissez des points d'accès REST pour gérer l'upload et le téléchargement.
@RestController
@RequestMapping("/data")
public class FileOperationsController {
private final FileHandlingService fileService;
public FileOperationsController(FileHandlingService fileService) {
this.fileService = fileService;
}
@PostMapping("/upload-chunk")
public ResponseEntity<String> handleChunkUpload(HttpServletRequest request) throws Exception {
fileService.processChunkUpload(request);
return ResponseEntity.ok("Segment reçu");
}
@GetMapping("/simple-download")
public void performSimpleDownload(HttpServletRequest request, HttpServletResponse response) throws IOException {
fileService.executeSimpleDownload(request, response);
}
@GetMapping("/chunked-download")
public ResponseEntity<String> initiateChunkedDownload() throws IOException {
fileService.startChunkedDownload();
return ResponseEntity.ok("Téléchargement fragmenté démarré");
}
}
Entités de données pour le suivi des fichiers
Créez des classes pour encapsuler les métadonnées des fichiers.
@Data
@AllArgsConstructor
@NoArgsConstructor
public class FileMetadata {
private long totalSize;
private String originalName;
}
@Data
public class ChunkInfo {
private String fileName;
private int segmentIndex;
private int totalSegments;
}
@Data
@AllArgsConstructor
@NoArgsConstructor
public class DownloadState {
private long fileSize;
private long rangeStart;
private long rangeEnd;
private long contentLength;
private String contentRangeHeader;
}
Service de logique métier
Implémentez la logique principale pour la fragmentation, la reprise et la fusion des fichiers.
@Service
public class FileHandlingService {
private static final String ENCODING = "UTF-8";
private final String uploadDirectory = Paths.get(System.getProperty("user.dir"), "file-storage", "uploads").toString();
private final String downloadDirectory = Paths.get(System.getProperty("user.dir"), "file-storage", "downloads").toString();
private static final long SEGMENT_SIZE = 50 * 1024 * 1024L; // 50 Mo par segment
private final ExecutorService threadPool = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors() * 2);
public void processChunkUpload(HttpServletRequest request) throws Exception {
ServletFileUpload uploadHandler = createUploadHandler();
List<FileItem> items = uploadHandler.parseRequest(request);
ChunkInfo chunkData = extractChunkMetadata(items);
saveSegment(items, chunkData);
if (chunkData.getSegmentIndex() == chunkData.getTotalSegments() - 1) {
assembleSegments(chunkData);
}
}
private ServletFileUpload createUploadHandler() {
DiskFileItemFactory factory = new DiskFileItemFactory();
factory.setSizeThreshold(4096);
File uploadDir = new File(uploadDirectory);
if (!uploadDir.exists()) uploadDir.mkdirs();
factory.setRepository(uploadDir);
ServletFileUpload upload = new ServletFileUpload(factory);
upload.setFileSizeMax(2L * 1024 * 1024 * 1024);
upload.setSizeMax(20L * 1024 * 1024 * 1024);
return upload;
}
private ChunkInfo extractChunkMetadata(List<FileItem> items) throws UnsupportedEncodingException {
ChunkInfo info = new ChunkInfo();
for (FileItem item : items) {
if (item.isFormField()) {
if ("segmentNumber".equals(item.getFieldName())) {
info.setSegmentIndex(Integer.parseInt(item.getString(ENCODING)));
} else if ("totalSegments".equals(item.getFieldName())) {
info.setTotalSegments(Integer.parseInt(item.getString(ENCODING)));
} else if ("name".equals(item.getFieldName())) {
info.setFileName(item.getString(ENCODING));
}
}
}
return info;
}
private void saveSegment(List<FileItem> items, ChunkInfo chunkData) throws Exception {
for (FileItem item : items) {
if (!item.isFormField()) {
String segmentName = chunkData.getSegmentIndex() + "_" + chunkData.getFileName();
File segmentFile = new File(uploadDirectory, segmentName);
if (!segmentFile.exists()) {
item.write(segmentFile);
}
}
}
}
private void assembleSegments(ChunkInfo chunkData) throws IOException, InterruptedException {
File outputFile = new File(uploadDirectory, chunkData.getFileName());
try (BufferedOutputStream output = new BufferedOutputStream(new FileOutputStream(outputFile))) {
for (int i = 0; i < chunkData.getTotalSegments(); i++) {
File segmentFile = new File(uploadDirectory, i + "_" + chunkData.getFileName());
while (!segmentFile.exists()) {
Thread.sleep(200);
}
byte[] data = Files.readAllBytes(segmentFile.toPath());
output.write(data);
output.flush();
segmentFile.delete();
}
}
}
public void executeSimpleDownload(HttpServletRequest request, HttpServletResponse response) throws IOException {
File targetFile = new File(downloadDirectory, "example.dat");
DownloadState state = computeDownloadState(targetFile.length(), request);
configureResponseHeaders(response, targetFile.getName(), state);
try (InputStream fileInput = new BufferedInputStream(new FileInputStream(targetFile));
OutputStream responseOutput = new BufferedOutputStream(response.getOutputStream())) {
fileInput.skip(state.getRangeStart());
byte[] buffer = new byte[8192];
long bytesTransferred = 0;
while (bytesTransferred < state.getContentLength()) {
int bytesRead = fileInput.read(buffer, 0, (int) Math.min(buffer.length, state.getContentLength() - bytesTransferred));
if (bytesRead == -1) break;
responseOutput.write(buffer, 0, bytesRead);
bytesTransferred += bytesRead;
}
}
}
private DownloadState computeDownloadState(long fileSize, HttpServletRequest request) {
long start = 0;
long end = fileSize - 1;
String rangeHeader = request.getHeader("Range");
if (rangeHeader != null && rangeHeader.startsWith("bytes=")) {
String[] ranges = rangeHeader.substring(6).split("-");
start = Long.parseLong(ranges[0]);
if (ranges.length > 1 && !ranges[1].isEmpty()) {
end = Long.parseLong(ranges[1]);
if (end >= fileSize) end = fileSize - 1;
}
}
long length = end - start + 1;
String contentRange = "bytes " + start + "-" + end + "/" + fileSize;
return new DownloadState(fileSize, start, end, length, contentRange);
}
public void startChunkedDownload() throws IOException {
File storageDir = new File(downloadDirectory);
if (!storageDir.exists()) storageDir.mkdirs();
FileMetadata metadata = fetchFileMetadata();
if (metadata != null) {
long segmentsCount = metadata.getTotalSize() / SEGMENT_SIZE + (metadata.getTotalSize() % SEGMENT_SIZE > 0 ? 1 : 0);
for (long i = 0; i < segmentsCount; i++) {
long segStart = i * SEGMENT_SIZE;
long segEnd = Math.min((i + 1) * SEGMENT_SIZE - 1, metadata.getTotalSize() - 1);
threadPool.submit(new SegmentDownloadTask(segStart, segEnd, i, metadata.getOriginalName()));
}
}
}
private FileMetadata fetchFileMetadata() throws IOException {
HttpClient client = HttpClients.createDefault();
HttpGet request = new HttpGet("http://localhost:8020/data/simple-download");
request.setHeader("Range", "bytes=0-0");
HttpResponse response = client.execute(request);
long size = Long.parseLong(response.getFirstHeader("FileSize").getValue());
String name = URLDecoder.decode(response.getFirstHeader("FileName").getValue(), ENCODING);
EntityUtils.consume(response.getEntity());
return new FileMetadata(size, name);
}
private class SegmentDownloadTask implements Runnable {
private final long startByte;
private final long endByte;
private final long index;
private final String fileName;
SegmentDownloadTask(long start, long end, long idx, String name) {
this.startByte = start;
this.endByte = end;
this.index = idx;
this.fileName = name;
}
@Override
public void run() {
try {
HttpClient client = HttpClients.createDefault();
HttpGet request = new HttpGet("http://localhost:8020/data/simple-download");
request.setHeader("Range", "bytes=" + startByte + "-" + endByte);
HttpResponse response = client.execute(request);
File segmentFile = new File(downloadDirectory, index + ".part");
try (InputStream entityStream = response.getEntity().getContent();
FileOutputStream fileOutput = new FileOutputStream(segmentFile)) {
byte[] buffer = new byte[4096];
int bytesRead;
while ((bytesRead = entityStream.read(buffer)) != -1) {
fileOutput.write(buffer, 0, bytesRead);
}
}
EntityUtils.consume(response.getEntity());
// Vérifier si tous les segments sont prêts pour la fusion
if (isLastSegment(endByte, fileName)) {
mergeSegments(fileName, index);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
private boolean isLastSegment(long endByte, String fileName) throws IOException {
// Logique pour vérifier si c'est le dernier segment
return false; // Implémentation simplifiée
}
private void mergeSegments(String fileName, long totalSegments) throws IOException {
File outputFile = new File(downloadDirectory, fileName);
try (BufferedOutputStream output = new BufferedOutputStream(new FileOutputStream(outputFile))) {
for (long i = 0; i <= totalSegments; i++) {
File partFile = new File(downloadDirectory, i + ".part");
while (!partFile.exists() || (i < totalSegments && partFile.length() < SEGMENT_SIZE)) {
Thread.sleep(500);
}
byte[] data = Files.readAllBytes(partFile.toPath());
output.write(data);
output.flush();
partFile.delete();
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
private void configureResponseHeaders(HttpServletResponse response, String fileName, DownloadState state) throws UnsupportedEncodingException {
response.setContentType("application/octet-stream");
response.setHeader("Content-Disposition", "attachment; filename=\"" + URLEncoder.encode(fileName, ENCODING) + "\"");
response.setHeader("Accept-Ranges", "bytes");
response.setHeader("Content-Range", state.getContentRangeHeader());
response.setHeader("Content-Length", String.valueOf(state.getContentLength()));
}
}
Tests et validation
Lancez l'application Spring Boot et accédez à l'interface utilisateur via un navigateur à l'adresse http://localhost:8020/upload.html. Testez l'upload de fichiers fragmentés en utilisant le glisser-déposer ou la sélection maneulle. Vérifiez que les segments sont créés dans le répertoire d'upload et fusionnés correctement. Testez les téléchargements simples et fragmentés pour assurer la reprise après interruption.