backend

package
v1.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 9, 2026 License: Apache-2.0 Imports: 26 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// this is the media type for directories in AWS and Nextcloud
	DirContentType     = "application/x-directory"
	DefaultContentType = "binary/octet-stream"

	// this is the minimum allowed size for mp parts
	MinPartSize = 5 * 1024 * 1024
)

Variables

View Source
var ErrSkipObj = errors.New("skip this object")

Functions

func AreEtagsSame added in v1.0.14

func AreEtagsSame(e1, e2 string) bool

AreEtagsSame compares 2 etags by ignoring quotes

func EvaluateMatchPreconditions added in v1.0.18

func EvaluateMatchPreconditions(etag string, ifMatch, ifNoneMatch *string) error

EvaluateMatchPreconditions evaluates if-match and if-none-match preconditions

func EvaluateObjectDeletePreconditions added in v1.0.18

func EvaluateObjectDeletePreconditions(etag string, modTime time.Time, size int64, preconditions ObjectDeletePreconditions) error

EvaluateObjectDeletePreconditions evaluates preconditions for DeleteObject

func EvaluateObjectPutPreconditions added in v1.1.0

func EvaluateObjectPutPreconditions(etag string, ifMatch, ifNoneMatch *string, objExists bool) error

EvaluateObjectPutPreconditions evaluates if-match and if-none-match preconditions for object PUT(PutObject, CompleteMultipartUpload) actions

func EvaluatePreconditions added in v1.0.18

func EvaluatePreconditions(etag string, modTime time.Time, preconditions PreConditions) error

EvaluatePreconditions takes the object ETag, the last modified time and evaluates the read preconditions: - if-match, - if-none-match - if-modified-since - if-unmodified-since if-match and if-none-match are ETag comparisions if-modified-since and if-unmodified-since are last modifed time comparisons

func GenerateEtag added in v1.0.14

func GenerateEtag(h hash.Hash) string

GenerateEtag generates a new quoted etag from the provided hash.Hash

func GetMultipartMD5

func GetMultipartMD5(parts []types.CompletedPart) string

func GetPtrFromString added in v1.0.8

func GetPtrFromString(str string) *string

func GetStringFromPtr added in v1.0.9

func GetStringFromPtr(str *string) string

func GetTimePtr

func GetTimePtr(t time.Time) *time.Time

func IsValidBucketName

func IsValidBucketName(name string) bool

func IsValidDirectoryName added in v1.0.19

func IsValidDirectoryName(name string) bool

IsValidDirectoryName returns true if the string is a valid name for a directory

func MkdirAll

func MkdirAll(path string, uid, gid int, doChown bool, dirPerm fs.FileMode) error

MkdirAll is similar to os.MkdirAll but it will return ErrObjectParentIsFile when appropriate MkdirAll creates a directory named path, along with any necessary parents, and returns nil, or else returns an error. The permission bits perm (before umask) are used for all directories that MkdirAll creates. Any newly created directory is set to provided uid/gid ownership. If path is already a directory, MkdirAll does nothing and returns nil. Any directory created will be set to provided uid/gid ownership if doChown is true.

func MoveFile added in v1.0.13

func MoveFile(source, destination string, perm os.FileMode) error

MoveFile moves a file from source to destination.

func ParseCopySource added in v1.0.8

func ParseCopySource(copySourceHeader string) (string, string, string, error)

ParseCopySource parses x-amz-copy-source header and returns source bucket, source object, versionId, error respectively

func ParseCopySourceRange added in v1.0.11

func ParseCopySourceRange(size int64, acceptRange string) (int64, int64, error)

ParseCopySourceRange parses input range header and returns startoffset, length and error. If no endoffset specified, then length is set to the object size

func ParseCreateBucketTags added in v1.0.19

func ParseCreateBucketTags(tagging []types.Tag) (map[string]string, error)

ParseCreateBucketTags parses and validates the bucket tagging from CreateBucket input

func ParseObjectRange added in v1.0.14

func ParseObjectRange(size int64, acceptRange string) (int64, int64, bool, error)

ParseObjectRange parses input range header and returns startoffset, length, isValid and error. If no endoffset specified, then length is set to the object size for invalid inputs, it returns no error, but isValid=false `InvalidRange` error is returnd, only if startoffset is greater than the object size

func ParseObjectTags added in v1.0.12

func ParseObjectTags(tagging string) (map[string]string, error)

ParseObjectTags parses the url encoded input string into map[string]string with unescaped key/value pair

func TrimEtag added in v1.0.11

func TrimEtag(etag *string) *string

func WalkDir added in v1.2.0

func WalkDir(fsys fs.FS, root string, fn WalkDirFunc) error

WalkDir walks the file tree rooted at root, calling fn for each file or directory in the tree, including root.

All errors that arise visiting files and directories are filtered by fn: see the fs.WalkDirFunc documentation for details.

The files are walked in lexical order, which makes the output deterministic but requires WalkDir to read an entire directory into memory before proceeding to walk that directory.

WalkDir does not follow symbolic links found in directories, but if root itself is a symbolic link, its target will be walked.

Types

type Backend

type Backend interface {
	fmt.Stringer
	Shutdown()

	// bucket operations
	ListBuckets(context.Context, s3response.ListBucketsInput) (s3response.ListAllMyBucketsResult, error)
	HeadBucket(context.Context, *s3.HeadBucketInput) (*s3.HeadBucketOutput, error)
	GetBucketAcl(context.Context, *s3.GetBucketAclInput) ([]byte, error)
	CreateBucket(_ context.Context, _ *s3.CreateBucketInput, defaultACL []byte) error
	PutBucketAcl(_ context.Context, bucket string, data []byte) error
	DeleteBucket(_ context.Context, bucket string) error
	PutBucketVersioning(_ context.Context, bucket string, status types.BucketVersioningStatus) error
	GetBucketVersioning(_ context.Context, bucket string) (s3response.GetBucketVersioningOutput, error)
	PutBucketPolicy(_ context.Context, bucket string, policy []byte) error
	GetBucketPolicy(_ context.Context, bucket string) ([]byte, error)
	DeleteBucketPolicy(_ context.Context, bucket string) error
	PutBucketOwnershipControls(_ context.Context, bucket string, ownership types.ObjectOwnership) error
	GetBucketOwnershipControls(_ context.Context, bucket string) (types.ObjectOwnership, error)
	DeleteBucketOwnershipControls(_ context.Context, bucket string) error
	PutBucketCors(_ context.Context, bucket string, cors []byte) error
	GetBucketCors(_ context.Context, bucket string) ([]byte, error)
	DeleteBucketCors(_ context.Context, bucket string) error

	// multipart operations
	CreateMultipartUpload(context.Context, s3response.CreateMultipartUploadInput) (s3response.InitiateMultipartUploadResult, error)
	CompleteMultipartUpload(context.Context, *s3.CompleteMultipartUploadInput) (_ s3response.CompleteMultipartUploadResult, versionid string, _ error)
	AbortMultipartUpload(context.Context, *s3.AbortMultipartUploadInput) error
	ListMultipartUploads(context.Context, *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResult, error)
	ListParts(context.Context, *s3.ListPartsInput) (s3response.ListPartsResult, error)
	UploadPart(context.Context, *s3.UploadPartInput) (*s3.UploadPartOutput, error)
	UploadPartCopy(context.Context, *s3.UploadPartCopyInput) (s3response.CopyPartResult, error)

	// standard object operations
	PutObject(context.Context, s3response.PutObjectInput) (s3response.PutObjectOutput, error)
	HeadObject(context.Context, *s3.HeadObjectInput) (*s3.HeadObjectOutput, error)
	GetObject(context.Context, *s3.GetObjectInput) (*s3.GetObjectOutput, error)
	GetObjectAcl(context.Context, *s3.GetObjectAclInput) (*s3.GetObjectAclOutput, error)
	GetObjectAttributes(context.Context, *s3.GetObjectAttributesInput) (s3response.GetObjectAttributesResponse, error)
	CopyObject(context.Context, s3response.CopyObjectInput) (s3response.CopyObjectOutput, error)
	ListObjects(context.Context, *s3.ListObjectsInput) (s3response.ListObjectsResult, error)
	ListObjectsV2(context.Context, *s3.ListObjectsV2Input) (s3response.ListObjectsV2Result, error)
	DeleteObject(context.Context, *s3.DeleteObjectInput) (*s3.DeleteObjectOutput, error)
	DeleteObjects(context.Context, *s3.DeleteObjectsInput) (s3response.DeleteResult, error)
	PutObjectAcl(context.Context, *s3.PutObjectAclInput) error
	ListObjectVersions(context.Context, *s3.ListObjectVersionsInput) (s3response.ListVersionsResult, error)

	// special case object operations
	RestoreObject(context.Context, *s3.RestoreObjectInput) error
	SelectObjectContent(ctx context.Context, input *s3.SelectObjectContentInput) func(w *bufio.Writer)

	// bucket tagging operations
	GetBucketTagging(_ context.Context, bucket string) (map[string]string, error)
	PutBucketTagging(_ context.Context, bucket string, tags map[string]string) error
	DeleteBucketTagging(_ context.Context, bucket string) error

	// object tagging operations
	GetObjectTagging(_ context.Context, bucket, object, versionId string) (map[string]string, error)
	PutObjectTagging(_ context.Context, bucket, object, versionId string, tags map[string]string) error
	DeleteObjectTagging(_ context.Context, bucket, object, versionId string) error

	// object lock operations
	PutObjectLockConfiguration(_ context.Context, bucket string, config []byte) error
	GetObjectLockConfiguration(_ context.Context, bucket string) ([]byte, error)
	PutObjectRetention(_ context.Context, bucket, object, versionId string, retention []byte) error
	GetObjectRetention(_ context.Context, bucket, object, versionId string) ([]byte, error)
	PutObjectLegalHold(_ context.Context, bucket, object, versionId string, status bool) error
	GetObjectLegalHold(_ context.Context, bucket, object, versionId string) (*bool, error)

	// non AWS actions
	ChangeBucketOwner(_ context.Context, bucket, owner string) error
	ListBucketsAndOwners(context.Context) ([]s3response.Bucket, error)
}

func New

func New() Backend

type BackendUnsupported

type BackendUnsupported struct{}

func (BackendUnsupported) AbortMultipartUpload

func (BackendUnsupported) ChangeBucketOwner

func (BackendUnsupported) ChangeBucketOwner(_ context.Context, bucket, owner string) error

func (BackendUnsupported) CreateBucket

func (BackendUnsupported) DeleteBucket

func (BackendUnsupported) DeleteBucket(_ context.Context, bucket string) error

func (BackendUnsupported) DeleteBucketCors added in v1.0.13

func (BackendUnsupported) DeleteBucketCors(_ context.Context, bucket string) error

func (BackendUnsupported) DeleteBucketOwnershipControls added in v1.0.3

func (BackendUnsupported) DeleteBucketOwnershipControls(_ context.Context, bucket string) error

func (BackendUnsupported) DeleteBucketPolicy

func (BackendUnsupported) DeleteBucketPolicy(_ context.Context, bucket string) error

func (BackendUnsupported) DeleteBucketTagging

func (BackendUnsupported) DeleteBucketTagging(_ context.Context, bucket string) error

func (BackendUnsupported) DeleteObject

func (BackendUnsupported) DeleteObjectTagging

func (BackendUnsupported) DeleteObjectTagging(_ context.Context, bucket, object, versionId string) error

func (BackendUnsupported) DeleteObjects

func (BackendUnsupported) GetBucketAcl

func (BackendUnsupported) GetBucketCors added in v1.0.13

func (BackendUnsupported) GetBucketCors(_ context.Context, bucket string) ([]byte, error)

func (BackendUnsupported) GetBucketOwnershipControls added in v1.0.3

func (BackendUnsupported) GetBucketOwnershipControls(_ context.Context, bucket string) (types.ObjectOwnership, error)

func (BackendUnsupported) GetBucketPolicy

func (BackendUnsupported) GetBucketPolicy(_ context.Context, bucket string) ([]byte, error)

func (BackendUnsupported) GetBucketTagging

func (BackendUnsupported) GetBucketTagging(_ context.Context, bucket string) (map[string]string, error)

func (BackendUnsupported) GetBucketVersioning

func (BackendUnsupported) GetObject

func (BackendUnsupported) GetObjectAcl

func (BackendUnsupported) GetObjectLegalHold

func (BackendUnsupported) GetObjectLegalHold(_ context.Context, bucket, object, versionId string) (*bool, error)

func (BackendUnsupported) GetObjectLockConfiguration

func (BackendUnsupported) GetObjectLockConfiguration(_ context.Context, bucket string) ([]byte, error)

func (BackendUnsupported) GetObjectRetention

func (BackendUnsupported) GetObjectRetention(_ context.Context, bucket, object, versionId string) ([]byte, error)

func (BackendUnsupported) GetObjectTagging

func (BackendUnsupported) GetObjectTagging(_ context.Context, bucket, object, versionId string) (map[string]string, error)

func (BackendUnsupported) HeadBucket

func (BackendUnsupported) HeadObject

func (BackendUnsupported) ListBucketsAndOwners

func (BackendUnsupported) ListBucketsAndOwners(context.Context) ([]s3response.Bucket, error)

func (BackendUnsupported) ListParts

func (BackendUnsupported) PutBucketAcl

func (BackendUnsupported) PutBucketAcl(_ context.Context, bucket string, data []byte) error

func (BackendUnsupported) PutBucketCors added in v1.0.13

func (BackendUnsupported) PutBucketCors(context.Context, string, []byte) error

func (BackendUnsupported) PutBucketOwnershipControls added in v1.0.3

func (BackendUnsupported) PutBucketOwnershipControls(_ context.Context, bucket string, ownership types.ObjectOwnership) error

func (BackendUnsupported) PutBucketPolicy

func (BackendUnsupported) PutBucketPolicy(_ context.Context, bucket string, policy []byte) error

func (BackendUnsupported) PutBucketTagging

func (BackendUnsupported) PutBucketTagging(_ context.Context, bucket string, tags map[string]string) error

func (BackendUnsupported) PutBucketVersioning

func (BackendUnsupported) PutBucketVersioning(_ context.Context, bucket string, status types.BucketVersioningStatus) error

func (BackendUnsupported) PutObjectAcl

func (BackendUnsupported) PutObjectLegalHold

func (BackendUnsupported) PutObjectLegalHold(_ context.Context, bucket, object, versionId string, status bool) error

func (BackendUnsupported) PutObjectLockConfiguration

func (BackendUnsupported) PutObjectLockConfiguration(_ context.Context, bucket string, config []byte) error

func (BackendUnsupported) PutObjectRetention

func (BackendUnsupported) PutObjectRetention(_ context.Context, bucket, object, versionId string, retention []byte) error

func (BackendUnsupported) PutObjectTagging

func (BackendUnsupported) PutObjectTagging(_ context.Context, bucket, object, versionId string, tags map[string]string) error

func (BackendUnsupported) RestoreObject

func (BackendUnsupported) SelectObjectContent

func (BackendUnsupported) SelectObjectContent(ctx context.Context, input *s3.SelectObjectContentInput) func(w *bufio.Writer)

func (BackendUnsupported) Shutdown

func (BackendUnsupported) Shutdown()

func (BackendUnsupported) String

func (BackendUnsupported) String() string

func (BackendUnsupported) UploadPart

type ByBucketName

type ByBucketName []s3response.ListAllMyBucketsEntry

func (ByBucketName) Len

func (d ByBucketName) Len() int

func (ByBucketName) Less

func (d ByBucketName) Less(i, j int) bool

func (ByBucketName) Swap

func (d ByBucketName) Swap(i, j int)

type ByObjectName

type ByObjectName []types.Object

func (ByObjectName) Len

func (d ByObjectName) Len() int

func (ByObjectName) Less

func (d ByObjectName) Less(i, j int) bool

func (ByObjectName) Swap

func (d ByObjectName) Swap(i, j int)

type FileSectionReadCloser added in v1.0.3

type FileSectionReadCloser struct {
	R io.Reader
	F *os.File
}

func (*FileSectionReadCloser) Close added in v1.0.3

func (f *FileSectionReadCloser) Close() error

func (*FileSectionReadCloser) Read added in v1.0.3

func (f *FileSectionReadCloser) Read(p []byte) (int, error)

type GetObjFunc

type GetObjFunc func(path string, d fs.DirEntry) (s3response.Object, error)

type GetVersionsFunc added in v1.0.8

type GetVersionsFunc func(path, versionIdMarker string, pastVersionIdMarker *bool, availableObjCount int, d fs.DirEntry) (*ObjVersionFuncResult, error)

type ListMultipartUploadsPage added in v1.2.0

type ListMultipartUploadsPage struct {
	Uploads            []s3response.Upload
	CommonPrefixes     []s3response.CommonPrefix
	IsTruncated        bool
	NextKeyMarker      string
	NextUploadIDMarker string
}

ListMultipartUploadsPage is the lister output

func ListMultipartUploads added in v1.2.0

func ListMultipartUploads(uploads []s3response.Upload, prefix, delimiter, keyMarker, uploadIdMarker string, maxUploads int) (*ListMultipartUploadsPage, error)

ListMultipartUploads initializes a multipart upload lister and calls Run()

type MultipartUploadLister added in v1.2.0

type MultipartUploadLister struct {
	Uploads        []s3response.Upload
	Prefix         string
	Delimiter      string
	MaxUploads     int
	KeyMarker      string
	UploadIDMarker string
}

MultipartUploadLister emits a ListMultipartUploads-compatible page from an already-sorted, already prefix- and key-marker-filtered upload list.

Assumptions about input Uploads:

  • Sorted by (Key asc, Initiated asc)
  • Filtered by Prefix
  • Filtered to start strictly after key-marker when key-marker was provided.

func (*MultipartUploadLister) Run added in v1.2.0

Run validates marker constraints, then performs a single-pass list that: - collapses uploads into CommonPrefixes when delimiter is set - enforces MaxUploads over (Uploads + CommonPrefixes) - computes truncation and next markers

type ObjVersionFuncResult added in v1.0.8

type ObjVersionFuncResult struct {
	ObjectVersions      []s3response.ObjectVersion
	DelMarkers          []types.DeleteMarkerEntry
	NextVersionIdMarker string
	Truncated           bool
}

type ObjectDeletePreconditions added in v1.0.18

type ObjectDeletePreconditions struct {
	IfMatch            *string
	IfMatchLastModTime *time.Time
	IfMatchSize        *int64
}

type PreConditions added in v1.0.18

type PreConditions struct {
	IfMatch       *string
	IfNoneMatch   *string
	IfModSince    *time.Time
	IfUnmodeSince *time.Time
}

type WalkDirFunc added in v1.2.0

type WalkDirFunc func(path string, d fs.DirEntry, err error) error

WalkDirFunc is the type of the function called by WalkDir to visit each file or directory.

The path argument contains the argument to WalkDir as a prefix. That is, if WalkDir is called with root argument "dir" and finds a file named "a" in that directory, the walk function will be called with argument "dir/a".

The d argument is the [DirEntry] for the named path.

The error result returned by the function controls how WalkDir continues. If the function returns the special value [SkipDir], WalkDir skips the current directory (path if d.IsDir() is true, otherwise path's parent directory). If the function returns the special value [SkipAll], WalkDir skips all remaining files and directories. Otherwise, if the function returns a non-nil error, WalkDir stops entirely and returns that error.

The err argument reports an error related to path, signaling that WalkDir will not walk into that directory. The function can decide how to handle that error; as described earlier, returning the error will cause WalkDir to stop walking the entire tree.

WalkDir calls the function with a non-nil err argument in two cases.

First, if the initial [Stat] on the root directory fails, WalkDir calls the function with path set to root, d set to nil, and err set to the error from fs.Stat.

Second, if a directory's ReadDir method (see [ReadDirFile]) fails, WalkDir calls the function with path set to the directory's path, d set to an [DirEntry] describing the directory, and err set to the error from ReadDir. In this second case, the function is called twice with the path of the directory: the first call is before the directory read is attempted and has err set to nil, giving the function a chance to return [SkipDir] or [SkipAll] and avoid the ReadDir entirely. The second call is after a failed ReadDir and reports the error from ReadDir. (If ReadDir succeeds, there is no second call.)

The differences between WalkDirFunc compared to path/filepath.WalkFunc are:

  • The second argument has type [DirEntry] instead of [FileInfo].
  • The function is called before reading a directory, to allow [SkipDir] or [SkipAll] to bypass the directory read entirely or skip all remaining files and directories respectively.
  • If a directory read fails, the function is called a second time for that directory to report the error.

type WalkResults

type WalkResults struct {
	CommonPrefixes []types.CommonPrefix
	Objects        []s3response.Object
	Truncated      bool
	NextMarker     string
}

func Walk

func Walk(ctx context.Context, fileSystem fs.FS, prefix, delimiter, marker string, max int32, getObj GetObjFunc, skipdirs []string) (WalkResults, error)

Walk walks the supplied fs.FS and returns results compatible with list objects responses

type WalkVersioningResults added in v1.0.8

type WalkVersioningResults struct {
	CommonPrefixes      []types.CommonPrefix
	ObjectVersions      []s3response.ObjectVersion
	DelMarkers          []types.DeleteMarkerEntry
	Truncated           bool
	NextMarker          string
	NextVersionIdMarker string
}

func WalkVersions added in v1.0.8

func WalkVersions(ctx context.Context, fileSystem fs.FS, prefix, delimiter, keyMarker, versionIdMarker string, max int, getObj GetVersionsFunc, skipdirs []string) (WalkVersioningResults, error)

WalkVersions walks the supplied fs.FS and returns results compatible with ListObjectVersions action response

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL