Fetch
in package
Class for holding and interacting with fetch.txt data.
Tags
Table of Contents
- FILENAME = "fetch.txt"
- The fetch filename.
- $bag : Bag
- The bag this fetch file is part of
- $curlOptions : array<string|int, mixed>
- Standard curl options to use.
- $curlVersion : string
- Curl version number string.
- $downloadQueue : array<string|int, mixed>
- Urls and Files that validated and should be downloaded.
- $fetchErrors : array<string|int, mixed>
- Errors
- $filename : string
- The current absolute path to the fetch.txt file.
- $files : array<string|int, mixed>
- Information from the fetch.txt, array of arrays with keys 'uri', 'size', and 'destination'
- __construct() : mixed
- Fetch constructor.
- addFile() : void
- Add a file to this fetch file.
- cleanup() : void
- Remove any downloaded files referenced in fetch.txt. This is called before we package up the Bag or finalize the directory.
- clearData() : void
- Clean up any downloaded files and then wipe the internal data array.
- download() : void
- Download a single file as it is added to the fetch file so we can generate checksums.
- downloadAll() : mixed
- Download the files.
- getData() : array<string|int, mixed>
- Return the array of file data.
- getErrors() : array<string|int, mixed>
- Return the errors.
- removeFile() : void
- Remove the URL (case-insensitive match) from the fetch file.
- reservedPath() : bool
- Check if the destination is supposed to be used by a fetched url.
- update() : void
- Update the fetch.txt on disk with the fetch file records.
- addError() : void
- Add an error for the fetch file.
- createCurl() : false|resource
- Initiate a cUrl handler
- createMultiCurl() : false|resource
- Create a cUrl multi handler.
- curlXferInfo() : int
- Compares current download size versus expected for cUrl progress.
- destinationExistsInFile() : bool
- Check if the destination path is already in the file.
- downloadFiles() : void
- Download files using Curl.
- internalValidateUrl() : bool
- BagItTools specific (non-spec) requirements for URLs.
- loadFiles() : void
- Load an existing fetch.txt
- resetErrors() : void
- Reset the error and warning logs.
- saveFileData() : void
- Write out data collected via curl to disk.
- setupCurl() : void
- Set general CURLOPTS based on the Curl version.
- urlExistsInFile() : bool
- Check if the url is already in the file.
- validateData() : mixed
- Validate fetch data.
- validateUrl() : bool
- Validate URLs can be processed by this library.
- writeToDisk() : void
- Utility to recreate the fetch file using the currently stored files.
Constants
FILENAME
The fetch filename.
private
mixed
FILENAME
= "fetch.txt"
Properties
$bag
The bag this fetch file is part of
private
Bag
$bag
$curlOptions
Standard curl options to use.
private
array<string|int, mixed>
$curlOptions
= [CURLOPT_CONNECTTIMEOUT => 10, CURLOPT_RETURNTRANSFER => true]
$curlVersion
Curl version number string.
private
string
$curlVersion
$downloadQueue
Urls and Files that validated and should be downloaded.
private
array<string|int, mixed>
$downloadQueue
= []
$fetchErrors
Errors
private
array<string|int, mixed>
$fetchErrors
= []
$filename
The current absolute path to the fetch.txt file.
private
string
$filename
$files
Information from the fetch.txt, array of arrays with keys 'uri', 'size', and 'destination'
private
array<string|int, mixed>
$files
Methods
__construct()
Fetch constructor.
public
__construct(Bag $bag[, bool $load = false ]) : mixed
Parameters
- $bag : Bag
-
The bag this fetch is part of.
- $load : bool = false
-
Whether to load a fetch.txt
Tags
Return values
mixed —addFile()
Add a file to this fetch file.
public
addFile(string $url, string $destination[, int|null $size = null ]) : void
Parameters
- $url : string
-
The remote URL for the file.
- $destination : string
-
The bag destination path for the file.
- $size : int|null = null
-
The expected size of the file, or null for unknown.
Tags
Return values
void —cleanup()
Remove any downloaded files referenced in fetch.txt. This is called before we package up the Bag or finalize the directory.
public
cleanup() : void
Tags
Return values
void —clearData()
Clean up any downloaded files and then wipe the internal data array.
public
clearData() : void
Tags
Return values
void —download()
Download a single file as it is added to the fetch file so we can generate checksums.
public
download(array<string|int, mixed> $fetchData) : void
Parameters
- $fetchData : array<string|int, mixed>
-
Array of data with keys 'uri', 'destination' and optionally 'size'.
Tags
Return values
void —downloadAll()
Download the files.
public
downloadAll() : mixed
Tags
Return values
mixed —getData()
Return the array of file data.
public
getData() : array<string|int, mixed>
Return values
array<string|int, mixed> —getErrors()
Return the errors.
public
getErrors() : array<string|int, mixed>
Return values
array<string|int, mixed> —Array of errors.
removeFile()
Remove the URL (case-insensitive match) from the fetch file.
public
removeFile(string $url) : void
Parameters
- $url : string
-
The url to remove.
Tags
Return values
void —reservedPath()
Check if the destination is supposed to be used by a fetched url.
public
reservedPath(string $dest) : bool
Parameters
- $dest : string
-
The relative path to check.
Return values
bool —True if the destination is in the fetch.txt
update()
Update the fetch.txt on disk with the fetch file records.
public
update() : void
Tags
Return values
void —addError()
Add an error for the fetch file.
private
addError(string $message) : void
Parameters
- $message : string
-
The message.
Return values
void —createCurl()
Initiate a cUrl handler
private
createCurl(string $url[, bool $single = false ][, int|null $size = null ]) : false|resource
Parameters
- $url : string
-
The URL to download.
- $single : bool = false
-
If this is a download() call versus a downloadAll() call.
- $size : int|null = null
-
Expected download size or null if unknown
Return values
false|resource —False on error, otherwise the cUl resource.
createMultiCurl()
Create a cUrl multi handler.
private
createMultiCurl() : false|resource
Return values
false|resource —False on error, otherwise the cUrl resource
curlXferInfo()
Compares current download size versus expected for cUrl progress.
private
static curlXferInfo(int $expectDl, int $currDl) : int
Parameters
- $expectDl : int
-
The expected download size (bytes).
- $currDl : int
-
The current download size (bytes).
Return values
int —1 if current download size is greater than 105% of the expected size.
destinationExistsInFile()
Check if the destination path is already in the file.
private
destinationExistsInFile(string $dest) : bool
Parameters
- $dest : string
-
Relative path to the destination file.
Return values
bool —True if a duplicate.
downloadFiles()
Download files using Curl.
private
downloadFiles() : void
Tags
Return values
void —internalValidateUrl()
BagItTools specific (non-spec) requirements for URLs.
private
internalValidateUrl(string $url) : bool
Parameters
- $url : string
-
The URL.
Return values
bool —True if we can process it.
loadFiles()
Load an existing fetch.txt
private
loadFiles() : void
Tags
Return values
void —resetErrors()
Reset the error and warning logs.
private
resetErrors() : void
Return values
void —saveFileData()
Write out data collected via curl to disk.
private
saveFileData(string $content, string $destination) : void
Parameters
- $content : string
-
The content from curl.
- $destination : string
-
The relative path to the final file.
Tags
Return values
void —setupCurl()
Set general CURLOPTS based on the Curl version.
private
setupCurl() : void
Return values
void —urlExistsInFile()
Check if the url is already in the file.
private
urlExistsInFile(string $url) : bool
Parameters
- $url : string
-
The url to check
Return values
bool —True if a duplicate.
validateData()
Validate fetch data.
private
validateData(array<string|int, mixed> $fetchData) : mixed
Parameters
- $fetchData : array<string|int, mixed>
-
Array with mandatory keys 'uri' and 'destination' and optional key 'size'.
Tags
Return values
mixed —validateUrl()
Validate URLs can be processed by this library.
private
validateUrl(string $url) : bool
Parameters
- $url : string
-
The URL.
Return values
bool —True if we can process it.
writeToDisk()
Utility to recreate the fetch file using the currently stored files.
private
writeToDisk() : void