- When you create a program to load,να γίνει υποκατηγορίαfrom
- Add your loader totablein this document
- do not addbin to store the new object
- to applyimmediate loading
- Whether you need to process your transfers is up to youwhere to do
- CarrierWave Uploader
- GitLab changes to CarrierWave
Where should you store your files?
The CarrierWave uploader determines where the files are stored. When you create a new Uploader class, you decide where to store the files for your new functionality.
First, ask yourself if you need a new Uploader class. The same Uploader class can be used for different attachment points or different models.
If you really want or need your own Uploader class, you should create onesubcategory of
Attachment loaderYou then inherit the storage location and directory schema from this class. The directory format is:
document.Participation(Model.class.tone, install_as.to_s, Model.ID card.to_s)
If you look at the GitLab repository, you will see that many users have their own repository. For item storage, this means loaders have their own containers. right now wethey do not encourageNew containers are added for the following reasons:
- Using a new container increases development time because you have to addGDK,GitLab for everyoneIcompressed natural gas.
- Using the new containers required changes to the GitLab.com infrastructure, which slowed down the release of new features
- Using a new container slows down the adoption of new features for self-managed GitLab installations: users can't start using your new features until your local GitLab admin configures the new container.
By using your existing barrel, you avoid all that extra work and friction. this
Gitlab.config.uploadsstorage location, what is it
Attachment loaderUse, make sure it is set.
Implement live streaming support
Below we describe how to achieve thisimmediate loadingsupport.
Using direct streaming isn't always necessary, but it's usually a good idea. Unless your function handles infrequent and small loads, you probably want to apply loads directly. Project avatars are an example of a feature with small and infrequent uploads: these avatars rarely change and their application imposes a strict size limit.
If your operation handles transfers that are neither frequent nor small, not implementing direct transfer support means you're taking on technical debt. At least, you should make surecapableAdd live streaming support later.
To support live streaming, you need two things:
- Pre-authorization endpoint in Rails
- Main routing rules
Workhorse does not know where to store your uploads. To find out, submit a pre-approval request. He also does not know if and where to apply for pre-approval. For this you need routing rules.
A note for those of us who remember,Workhorse was a separate project: It is no longer necessary to split these two steps into separate merge requests. In fact, it may be easier to do both in one merge request.
Add Workhorse routing rules
Routing rules are defined inworkhorse/internal/upstream/routes.go.They include:
- HTTP verb (usually "POST" or "PUT")
- path regular expression
- Transport type: Multipart MIME or "full request body"
- Alternatively, you can also assign HTTP headers, such as
vessel.ruta("tan", way project api+"package/nuget/"., mimeMultipartUploader),
You should add a test to the routing rule
Try expedited shippingexistsmain/upload_test.go.
You should also manually verify that Workhorse issues a pre-approval request when you submit a request to load a new feature. You can verify this by looking at the Rails access log. This is necessary because if you get the routing rules wrong, you won't have a hard failure: you'll just end up using a less efficient default route.
Add an endpoint before authorization
We distinguish three cases: Rails controllers, Grape API endpoints, and GraphQL resources.
First the bad news: direct transfers in GraphQL are not currently supported. This is because Workhorse does not parse GraphQL queries. See alsoIssue #280819.Consider accepting the file upload via Grape.
For the Grape preauthorization endpoint, look for existing example implementations
/approveRoute. An example ispostal
:id/upload/authorizationendpoint.This example uses a FileUloader, which means that the upload is stored in the storage location (bucket) of this Uploader class.
For Rails endpoints you can useThe main authorization problem.
Some features require us to manage uploads, such as extracting metadata from uploaded files. There are several different ways to achieve this. The main options areWhereProcessing implementation, i.e. "who is the processor".
|processor||Can it be loaded directly?||Can an HTTP request be rejected?||to apply|
Handling in Rails looks attractive, but it usually causes scaling problems because you can't use transfers directly. You will then be forced to rebuild your function by doing so in Workhorse. So, if your operational requirements allow it, running on Sidekiq can be a good balance between complexity and scalability.
GitLab uses a modified versionCarrierTransfer management. Below we describe how we use CarrierWave and how we modify it.
The basic idea of CarrierWave isUploaded by userclass. Loaders define where the files are stored and can optionally contain validation and processing logic. To use a loader, you must bind it to a text column in the ActiveRecord model. This is called "installation" and the column is named
placement point.For example:
class Work < application file install the uploader : avatar, Attachment loaderend
Now if you upload a file named
civet cat.pngThe idea is that in
project.avatarcolumn, CarrierWave storage array
civet cat.png, and the AttachmentUploader class contains configuration data and a directory schema. For example, if the project ID is 123, the actual file might be located at
/var/opt/gitlab/gitlab-rails/uploads/-/system/project/avatar/123/It is up to the loader to use additional configuration options (
/var/opt/gitlab/gitlab-rails/uploads), Model Name (
Work), model ID (
123) and support point (
The sender specifies a personal directory to store your uploads. this
placement pointA column in the model contains the file name.
you never visit
placement pointDirect because CarrierWave defines a getter and setter on your model that operates on the file handler object.
Optional loader behavior
In addition to specifying the storage directory to upload to, the CarrierWave Uploader can implement many other behaviors through callbacks. Not all of these behaviors are available in GitLab. Specifically, you can't use it right now
VersionCarrier wave mechanism. Things you can do include:
- file name check
- Not compatible with live streaming:Perform pre-processing on the file contents, such as resizing the image
- Not compatible with live streaming:encryption at rest
CarrierWave's preprocessing behavior, such as image resizing or encryption, requires local access to the uploaded files. This forces you to load the edited files from Ruby. This is opposed to direct loading, which is approxYesLoading in Ruby. If you use direct loading with a loader that has preprocessing behavior, the preprocessing behavior will be silently skipped.
CarrierWave data storage mechanism
CarrierWave has 2 data storage mechanisms:
|CarrierWave class||GitLab time||describe|
|Local files, accessible through Ruby|
|Files in the cloud, accessible viadental ornament|
GitLab uses both engines depending on the configuration.
A typical way to select a storage mechanism in CarrierWave is to use
uploader.storageclass method. At GitLab, we don't do that. we covered
Uploaded by #storageinstead of this. This allows us to modify the storage engine file by file.
CarrierWave file life cycle
The uploader is connected to two storage areas: normal storage and cache storage. Each has its own data storage mechanism. If a file is assigned a mount point setter (
project.avatar = File.open('/tmp/tanuki.png')) as a side effect, you have to pass
Crypt!method. To save the file, you need to call it somehow
Store!method. This either happens insideActiveRecord callbackor call
Store!in the case of the loader.
You usually don't need to use it
Store!But if you need to patch changes in GitLab CarrierWave, it's useful to know that it's there and always called. In particular, it is good to know the CarrierWave preprocessing behavior (
procedureetc) are implemented as
before: cachehooks, and in the case of direct loads, these hooks are ignored and will not work.
Direct transmission skips all CarrierWave
GitLab changes to CarrierWave
GitLab uses a modified version of CarrierWave to enable many things.
Data transfer between storage machines
existsapp/uploader/object_storage.rbThere is code to migrate user data between local storage and object storage. This code exists because GitLab.com has long saved commits to local storage via NFS. This changed when we had to move transports to object storage as part of an infrastructure migration.
SaveIt varies depending on the upload to GitLab and why we have database columns like
Direct upload via Workhorse
Workhorse direct loading is a mechanism that allows us to accept large loads without consuming too much Ruby CPU time. Workhorse is written in Go, and goroutines have much fewer resources than Rubythreads.
See how live streaming works.
- Workhorse accepts user upload requests
- Workhorse uses Rails to pre-validate requests and obtain temporary load positions
- Workhorse stores uploaded files from user requests in a temporary upload location
- Workhorse propagates the request to Rails
- Rails performs a remote copy function to copy an uploaded file from a temporary location to a final location
- Rails deletes the temporary transfer
- Workhorse flushes the temporary transfer a second time to prevent Rails from timing out
Crypt!return an instance
Store!afterwardUpload the file using Fog.
In the case of object storage, copying from a temporary location to a final location is accomplished with Rails tricking CarrierWave with GitLab-specific modifications. When CarrierWave tries
CarrierWave::Save::Fog::FileThe file handle for the temporary file. During this period
Store!Phase, CarrierWave and timecopythis file to the expected location.
The Scalability::Frameworks team makes saving and loading objects easier to use and more powerful. If you add or change loaders, it would help us to update this form as well. This helps us understand where and how transmitters are being used.
Characteristic container elements
|feature||loading technology||Uploaded by user||barrel structure|
|Production from the assembly line|
|Live job traces|
|Task tracking file|
|Autoscale cursor cache||Does not apply|
|backup||Does not apply|
|The file system of the Git language|
|design management file|
|Project Management Thumbnail|
|General file upload|
|General file transfer - individual fragments|
|Global layout settings|
|Package Management Resources (Non-NPM)|
|NPM Package Management Resources|
|Debian Package Management Resources|
|Rely on a proxy cache|
|Terraform state file|
|Website content archive|