diff --git a/docs/03-endpoints/api-v2/editing-values.md b/docs/03-endpoints/api-v2/editing-values.md index 4b13283203..84f7cf8575 100644 --- a/docs/03-endpoints/api-v2/editing-values.md +++ b/docs/03-endpoints/api-v2/editing-values.md @@ -302,10 +302,7 @@ For example, to get a JPG thumbnail image that is 150 pixels wide, you would add ### Upload Files to DSP-INGEST -Support for DSP-INGEST is in its early stage and currently mainly intended for ingesting large amounts of data. -When a file has been ingested through DSP-INGEST, -it is necessary to send the header `X-Asset-Ingested` -along with the request to create the file value resource in DSP-API. +Support for uploads with DSP-INGEST is now the preferred method of uploading files (and also for ingesting large amounts of data internally). ### Submit A File Value to DSP-API diff --git a/docs/05-internals/design/api-v2/sipi.md b/docs/05-internals/design/api-v2/sipi.md index 0565340e7f..f629e8539b 100644 --- a/docs/05-internals/design/api-v2/sipi.md +++ b/docs/05-internals/design/api-v2/sipi.md @@ -18,57 +18,9 @@ found in `sipi/scripts` in the DSP-API source tree. Each of these scripts expects a [JSON Web Token](https://jwt.io/) in the URL parameter `token`. In all cases, the token must be signed by DSP-API, -it must have an expiration date and not have expired, its issuer must equal -the hostname and port of the API, and its audience must include `Sipi`. +it must have an expiration date and not have expired, its issuer must equal +the hostname and port of the API, and its audience must include `Sipi`. The other contents of the expected tokens are described below. - -### upload.lua - -The `upload.lua` script is available at Sipi's `upload` route. It processes one -or more file uploads submitted to Sipi. It converts uploaded images to JPEG 2000 -format, and stores them in Sipi's `tmp` directory. The usage of this script is described in -[Upload Files to Sipi](../../../03-endpoints/api-v2/editing-values.md#upload-files-to-sipi). - -### upload_without_processing.lua - -The `upload_without_processing.lua` script is available at Sipi's `upload_without_processing` route. -It receives files submitted to Sipi but does not process them. -Instead, it stores them as is in Sipi's `tmp` directory. - -### store.lua - -The `store.lua` script is available at Sipi's `store` route. It moves a file -from temporary to permanent storage. It expects an HTTP `POST` request containing -`application/x-www-form-urlencoded` data with the parameters `prefix` (the -project shortcode) and `filename` (the internal Sipi-generated filename of the file -to be moved). - -The JWT sent to this script must contain the key `knora-data`, whose value -must be a JSON object containing: - -- `permission`: must be `StoreFile` -- `prefix`: the project shortcode submitted in the form data -- `filename`: the filename submitted in the form data - -### delete_temp_file.lua - -The `delete_temp_file.lua` script is available at Sipi's `delete_temp_file` route. -It is used only if DSP-API rejects a file value update request. It expects an -HTTP `DELETE` request, with a filename as the last component of the URL. - -The JWT sent to this script must contain the key `knora-data`, whose value -must be a JSON object containing: - -- `permission`: must be `DeleteTempFile` -- `filename`: must be the same as the filename submitted in the URL - -### clean_temp_dir.lua - -The `clean_temp_dir.lua` script is available at Sipi's `clean_temp_dir` route. -When called, it deletes old temporary files from `tmp` and (recursively) from any subdirectories. -The maximum allowed age of temporary files can be set in Sipi's configuration file, -using the parameter `max_temp_file_age`, which takes a value in seconds. - The `clean_temp_dir` route requires basic authentication. ## SipiConnector @@ -78,38 +30,3 @@ with Sipi. It blocks while processing each request, to ensure that the number of concurrent requests to Sipi is not greater than `akka.actor.deployment./storeManager/iiifManager/sipiConnector.nr-of-instances`. If it encounters an error, it returns `SipiException`. - -## The Image File Upload Workflow - -1. The client uploads an image file to the `upload` route, which runs - `upload.lua`. The image is converted to JPEG 2000 and stored in Sipi's `tmp` - directory. In the response, the client receives the JPEG 2000's unique, - randomly generated filename. -2. The client submits a JSON-LD request to a DSP-API route (`/v2/values` or `/v2/resources`) - to create or change a file value. The request includes Sipi's internal filename. -3. During parsing of this JSON-LD request, a `StillImageFileValueContentV2` - is constructed to represent the file value. During the construction of this - object, a `GetFileMetadataRequestV2` is sent to `SipiConnector`, which - uses Sipi's built-in `knora.json` route to get the rest of the file's - metadata. -4. A responder (`ResourcesResponderV2` or `ValuesResponderV2`) validates - the request and updates the triplestore. (If it is `ResourcesResponderV2`, - it asks `ValuesResponderV2` to generate SPARQL for the values.) -5. The responder that did the update calls `ValueUtilV2.doSipiPostUpdate`. - If the triplestore update was successful, this method sends - `MoveTemporaryFileToPermanentStorageRequestV2` to `SipiConnector`, which - makes a request to Sipi's `store` route. Otherwise, the same method sends - `DeleteTemporaryFileRequestV2` to `SipiConnector`, which makes a request - to Sipi's `delete_temp_file` route. - -If the request to DSP-API cannot be parsed, the temporary file is not deleted -immediately, but it will be deleted during the processing of a subsequent -request by Sipi's `upload` route. - -If Sipi's `store` route fails, DSP-API returns the `SipiException` to the client. -In this case, manual intervention may be necessary to restore consistency -between DSP-API and Sipi. - -If Sipi's `delete_temp_file` route fails, the error is not returned to the client, -because there is already a DSP-API error that needs to be returned to the client. -In this case, the Sipi error is simply logged. diff --git a/integration/src/test/resources/sipi.docker-config.lua b/integration/src/test/resources/sipi.docker-config.lua index 0efb43d681..88c35a4d42 100644 --- a/integration/src/test/resources/sipi.docker-config.lua +++ b/integration/src/test/resources/sipi.docker-config.lua @@ -181,16 +181,6 @@ fileserver = { -- Custom routes. Each route is an URL path associated with a Lua script. -- routes = { - { - method = 'POST', - route = '/upload', - script = 'upload.lua' - }, - { - method = 'POST', - route = '/store', - script = 'store.lua' - }, { method = 'DELETE', route = '/delete_temp_file', diff --git a/integration/src/test/scala/org/knora/sipi/SipiClientTestDelegator.scala b/integration/src/test/scala/org/knora/sipi/SipiClientTestDelegator.scala index eed65733d9..34d95938c2 100644 --- a/integration/src/test/scala/org/knora/sipi/SipiClientTestDelegator.scala +++ b/integration/src/test/scala/org/knora/sipi/SipiClientTestDelegator.scala @@ -10,11 +10,8 @@ import zio.ULayer import zio.ZLayer import zio.nio.file.Path -import org.knora.webapi.messages.store.sipimessages.DeleteTemporaryFileRequest -import org.knora.webapi.messages.store.sipimessages.MoveTemporaryFileToPermanentStorageRequest import org.knora.webapi.messages.store.sipimessages.SipiGetTextFileRequest import org.knora.webapi.messages.store.sipimessages.SipiGetTextFileResponse -import org.knora.webapi.messages.v2.responder.SuccessResponseV2 import org.knora.webapi.slice.admin.api.model.MaintenanceRequests.AssetId import org.knora.webapi.slice.admin.domain.model.KnoraProject import org.knora.webapi.slice.admin.domain.model.User @@ -45,15 +42,6 @@ case class SipiServiceTestDelegator( if (whichSipi.useLive) { live } else { mock } - /** - * Asks Sipi for metadata about a file in the tmp folder, served from the 'knora.json' route. - * - * @param filename the path to the file. - * @return a [[FileMetadataSipiResponse]] containing the requested metadata. - */ - override def getFileMetadataFromSipiTemp(filename: String): Task[FileMetadataSipiResponse] = - sipiService.getFileMetadataFromSipiTemp(filename) - /** * Asks DSP-Ingest for metadata about a file in permanent location, served from the 'knora.json' route. * @@ -67,26 +55,6 @@ case class SipiServiceTestDelegator( ): Task[FileMetadataSipiResponse] = sipiService.getFileMetadataFromDspIngest(shortcode, assetId) - /** - * Asks Sipi to move a file from temporary storage to permanent storage. - * - * @param moveTemporaryFileToPermanentStorageRequestV2 the request. - * @return a [[SuccessResponseV2]]. - */ - override def moveTemporaryFileToPermanentStorage( - moveTemporaryFileToPermanentStorageRequestV2: MoveTemporaryFileToPermanentStorageRequest, - ): Task[SuccessResponseV2] = - sipiService.moveTemporaryFileToPermanentStorage(moveTemporaryFileToPermanentStorageRequestV2) - - /** - * Asks Sipi to delete a temporary file. - * - * @param deleteTemporaryFileRequestV2 the request. - * @return a [[SuccessResponseV2]]. - */ - override def deleteTemporaryFile(deleteTemporaryFileRequestV2: DeleteTemporaryFileRequest): Task[SuccessResponseV2] = - sipiService.deleteTemporaryFile(deleteTemporaryFileRequestV2) - /** * Asks Sipi for a text file used internally by Knora. * diff --git a/integration/src/test/scala/org/knora/webapi/E2ESpec.scala b/integration/src/test/scala/org/knora/webapi/E2ESpec.scala index 35e64908ec..286a2975b0 100644 --- a/integration/src/test/scala/org/knora/webapi/E2ESpec.scala +++ b/integration/src/test/scala/org/knora/webapi/E2ESpec.scala @@ -151,8 +151,8 @@ abstract class E2ESpec protected def getResponseAsJsonLD(request: HttpRequest): JsonLDDocument = UnsafeZioRun.runOrThrow(ZIO.serviceWithZIO[TestClientService](_.getResponseJsonLD(request))) - protected def uploadToSipi(loginToken: String, filesToUpload: Seq[FileToUpload]): SipiUploadResponse = - UnsafeZioRun.runOrThrow(ZIO.serviceWithZIO[TestClientService](_.uploadToSipi(loginToken, filesToUpload))) + protected def uploadToIngest(loginToken: String, filesToUpload: Seq[FileToUpload]): SipiUploadResponse = + UnsafeZioRun.runOrThrow(ZIO.serviceWithZIO[TestClientService](_.uploadToIngest(loginToken, filesToUpload))) protected def responseToJsonLDDocument(httpResponse: HttpResponse): JsonLDDocument = { val responseBodyFuture: Future[String] = diff --git a/integration/src/test/scala/org/knora/webapi/ITKnoraLiveSpec.scala b/integration/src/test/scala/org/knora/webapi/ITKnoraLiveSpec.scala index 73122215b5..e7989052e5 100644 --- a/integration/src/test/scala/org/knora/webapi/ITKnoraLiveSpec.scala +++ b/integration/src/test/scala/org/knora/webapi/ITKnoraLiveSpec.scala @@ -171,28 +171,13 @@ abstract class ITKnoraLiveSpec .getOrThrowFiberFailure() } - protected def uploadToSipi(loginToken: String, filesToUpload: Seq[FileToUpload]): SipiUploadResponse = + protected def uploadToIngest(loginToken: String, filesToUpload: Seq[FileToUpload]): SipiUploadResponse = Unsafe.unsafe { implicit u => runtime.unsafe .run( for { testClient <- ZIO.service[TestClientService] - result <- testClient.uploadToSipi(loginToken, filesToUpload) - } yield result, - ) - .getOrThrow() - } - - protected def uploadWithoutProcessingToSipi( - loginToken: String, - filesToUpload: Seq[FileToUpload], - ): SipiUploadWithoutProcessingResponse = - Unsafe.unsafe { implicit u => - runtime.unsafe - .run( - for { - testClient <- ZIO.service[TestClientService] - result <- testClient.uploadWithoutProcessingToSipi(loginToken, filesToUpload) + result <- testClient.uploadToIngest(loginToken, filesToUpload) } yield result, ) .getOrThrow() diff --git a/integration/src/test/scala/org/knora/webapi/core/LayersTest.scala b/integration/src/test/scala/org/knora/webapi/core/LayersTest.scala index 34f6758b3e..f3b6db8b28 100644 --- a/integration/src/test/scala/org/knora/webapi/core/LayersTest.scala +++ b/integration/src/test/scala/org/knora/webapi/core/LayersTest.scala @@ -76,6 +76,7 @@ import org.knora.webapi.slice.shacl.api.ShaclApiModule import org.knora.webapi.store.iiif.IIIFRequestMessageHandler import org.knora.webapi.store.iiif.IIIFRequestMessageHandlerLive import org.knora.webapi.store.iiif.api.SipiService +import org.knora.webapi.store.iiif.impl.SipiServiceLive import org.knora.webapi.store.triplestore.api.TriplestoreService import org.knora.webapi.store.triplestore.impl.TriplestoreServiceLive import org.knora.webapi.store.triplestore.upgrade.RepositoryUpdater @@ -238,6 +239,7 @@ object LayersTest { ListsService.layer, IIIFRequestMessageHandlerLive.layer, IriService.layer, + SipiServiceLive.layer, ) private val commonLayersForAllIntegrationTests = diff --git a/integration/src/test/scala/org/knora/webapi/e2e/v2/ValuesV2R2RSpec.scala b/integration/src/test/scala/org/knora/webapi/e2e/v2/ValuesV2R2RSpec.scala index 15cc3dc5c2..f403559440 100644 --- a/integration/src/test/scala/org/knora/webapi/e2e/v2/ValuesV2R2RSpec.scala +++ b/integration/src/test/scala/org/knora/webapi/e2e/v2/ValuesV2R2RSpec.scala @@ -6,42 +6,33 @@ package org.knora.webapi.e2e.v2 import org.apache.pekko.http.scaladsl.model.HttpEntity +import org.apache.pekko.http.scaladsl.model.HttpResponse import org.apache.pekko.http.scaladsl.model.StatusCodes import org.apache.pekko.http.scaladsl.model.headers.BasicHttpCredentials -import org.apache.pekko.http.scaladsl.server.RouteConcatenation.* -import zio.ZIO import scala.concurrent.ExecutionContextExecutor -import dsp.errors.AssertionException import dsp.errors.BadRequestException import dsp.valueobjects.Iri import org.knora.webapi.* import org.knora.webapi.messages.IriConversions.* import org.knora.webapi.messages.OntologyConstants +import org.knora.webapi.messages.OntologyConstants.KnoraApiV2Complex as OntConsts import org.knora.webapi.messages.SmartIri import org.knora.webapi.messages.StringFormatter import org.knora.webapi.messages.store.triplestoremessages.RdfDataObject import org.knora.webapi.messages.util.rdf.* import org.knora.webapi.messages.util.search.SparqlQueryConstants -import org.knora.webapi.routing.UnsafeZioRun -import org.knora.webapi.routing.v2.ValuesRouteV2 import org.knora.webapi.sharedtestdata.SharedTestDataADM -import org.knora.webapi.slice.search.api.SearchApiRoutes import org.knora.webapi.util.MutableTestIri /** * Tests creating a still image file value using a mock Sipi. */ -class ValuesV2R2RSpec extends R2RSpec { +class ValuesV2R2RSpec extends ITKnoraLiveSpec { private implicit val stringFormatter: StringFormatter = StringFormatter.getGeneralInstance - private val valuesPath = ValuesRouteV2().makeRoute - private val searchPath = UnsafeZioRun - .runOrThrow(ZIO.serviceWith[SearchApiRoutes](_.routes)) - .reduce(_ ~ _) - implicit val ec: ExecutionContextExecutor = system.dispatcher private val aThingPictureIri = "http://rdfh.ch/0001/a-thing-picture" @@ -70,14 +61,14 @@ class ValuesV2R2RSpec extends R2RSpec { .getResourceWithSpecifiedProperties(resourceIri, propertyIrisForGravsearch) .toString() - // Run the query. - Post( - "/v2/searchextended", + val request = Post( + baseApiUrl + "/v2/searchextended", HttpEntity(SparqlQueryConstants.`application/sparql-query`, gravsearchQuery), - ) ~> addCredentials(BasicHttpCredentials(userEmail, password)) ~> searchPath ~> check { - assert(status == StatusCodes.OK, response.toString) - responseToJsonLDDocument(response) - } + ) ~> addCredentials(BasicHttpCredentials(userEmail, password)) + + val response: HttpResponse = singleAwaitingRequest(request) + assert(response.status == StatusCodes.OK) + responseToJsonLDDocument(response) } private def getValuesFromResource(resource: JsonLDDocument, propertyIriInResult: SmartIri): JsonLDArray = @@ -88,32 +79,14 @@ class ValuesV2R2RSpec extends R2RSpec { propertyIriInResult: SmartIri, expectedValueIri: IRI, ): JsonLDObject = { - val resourceIri: IRI = - resource.body.requireStringWithValidation(JsonLDKeywords.ID, validationFun) - val propertyValues: JsonLDArray = - getValuesFromResource(resource = resource, propertyIriInResult = propertyIriInResult) - - val matchingValues: Seq[JsonLDObject] = propertyValues.value.collect { - case jsonLDObject: JsonLDObject - if jsonLDObject.requireStringWithValidation( - JsonLDKeywords.ID, - validationFun, - ) == expectedValueIri => - jsonLDObject - } - - if (matchingValues.isEmpty) { - throw AssertionException( - s"Property <$propertyIriInResult> of resource <$resourceIri> does not have value <$expectedValueIri>", - ) - } - - if (matchingValues.size > 1) { - throw AssertionException( - s"Property <$propertyIriInResult> of resource <$resourceIri> has more than one value with the IRI <$expectedValueIri>", - ) - } + val matchingValues: Seq[JsonLDObject] = + getValuesFromResource(resource = resource, propertyIriInResult = propertyIriInResult).value.collect { + case jsonLDObject: JsonLDObject + if jsonLDObject.requireStringWithValidation(JsonLDKeywords.ID, validationFun) == expectedValueIri => + jsonLDObject + } + assert(matchingValues.size == 1) matchingValues.head } @@ -140,8 +113,7 @@ class ValuesV2R2RSpec extends R2RSpec { "The values v2 endpoint" should { "update a still image file value using a mock Sipi" in { val resourceIri: IRI = aThingPictureIri - val internalFilename = "IQUO3t1AABm-FSLC0vNvVpr.jp2" - + val internalFilename = "De6XyNL4H71-D9QxghOuOPJ.jp2" val jsonLDEntity = s"""{ | "@id" : "$resourceIri", @@ -157,36 +129,31 @@ class ValuesV2R2RSpec extends R2RSpec { | } |}""".stripMargin - Put("/v2/values", HttpEntity(RdfMediaTypes.`application/ld+json`, jsonLDEntity)) ~> addCredentials( - BasicHttpCredentials(anythingUserEmail, password), - ) ~> valuesPath ~> check { - assert(status == StatusCodes.OK, response.toString) - val responseJsonDoc = responseToJsonLDDocument(response) - val valueIri: IRI = - responseJsonDoc.body.requireStringWithValidation(JsonLDKeywords.ID, validationFun) - stillImageFileValueIri.set(valueIri) - val valueType: SmartIri = - responseJsonDoc.body.requireStringWithValidation(JsonLDKeywords.TYPE, stringFormatter.toSmartIriWithErr) - valueType should ===(OntologyConstants.KnoraApiV2Complex.StillImageFileValue.toSmartIri) - - val savedValue: JsonLDObject = getValue( - resourceIri = resourceIri, - propertyIriForGravsearch = OntologyConstants.KnoraApiV2Complex.HasStillImageFileValue.toSmartIri, - propertyIriInResult = OntologyConstants.KnoraApiV2Complex.HasStillImageFileValue.toSmartIri, - expectedValueIri = stillImageFileValueIri.get, - userEmail = anythingUserEmail, - ) - - savedValue - .getRequiredString(OntologyConstants.KnoraApiV2Complex.FileValueHasFilename) - .fold(msg => throw BadRequestException(msg), identity) should ===(internalFilename) - savedValue - .getRequiredInt(OntologyConstants.KnoraApiV2Complex.StillImageFileValueHasDimX) - .fold(e => throw BadRequestException(e), identity) should ===(512) - savedValue - .getRequiredInt(OntologyConstants.KnoraApiV2Complex.StillImageFileValueHasDimY) - .fold(e => throw BadRequestException(e), identity) should ===(256) - } + val request = + Put(baseApiUrl + "/v2/values", HttpEntity(RdfMediaTypes.`application/ld+json`, jsonLDEntity)) + ~> addCredentials(BasicHttpCredentials(anythingUserEmail, password)) + + val response: HttpResponse = singleAwaitingRequest(request) + assert(response.status == StatusCodes.OK) + val responseJsonDoc = responseToJsonLDDocument(response) + val valueIri: IRI = + responseJsonDoc.body.requireStringWithValidation(JsonLDKeywords.ID, validationFun) + stillImageFileValueIri.set(valueIri) + val valueType: SmartIri = + responseJsonDoc.body.requireStringWithValidation(JsonLDKeywords.TYPE, stringFormatter.toSmartIriWithErr) + valueType should ===(OntConsts.StillImageFileValue.toSmartIri) + + val savedValue: JsonLDObject = getValue( + resourceIri = resourceIri, + propertyIriForGravsearch = OntConsts.HasStillImageFileValue.toSmartIri, + propertyIriInResult = OntConsts.HasStillImageFileValue.toSmartIri, + expectedValueIri = stillImageFileValueIri.get, + userEmail = anythingUserEmail, + ) + + savedValue.getRequiredString(OntConsts.FileValueHasFilename).toOption should contain(internalFilename) + savedValue.getRequiredInt(OntConsts.StillImageFileValueHasDimX).toOption should contain(72) + savedValue.getRequiredInt(OntConsts.StillImageFileValueHasDimY).toOption should contain(72) } } } diff --git a/integration/src/test/scala/org/knora/webapi/it/v2/CopyrightAndLicensesSpec.scala b/integration/src/test/scala/org/knora/webapi/it/v2/CopyrightAndLicensesSpec.scala index 329911acec..74d5797c82 100644 --- a/integration/src/test/scala/org/knora/webapi/it/v2/CopyrightAndLicensesSpec.scala +++ b/integration/src/test/scala/org/knora/webapi/it/v2/CopyrightAndLicensesSpec.scala @@ -88,7 +88,7 @@ object CopyrightAndLicensesSpec extends E2EZSpec { val jsonLd = UploadFileRequest .make( FileType.StillImageFile(), - "internalFilename", + "internalFilename.jpg", copyrightAttribution = Some(copyrightAttribution), license = Some(license), ) diff --git a/integration/src/test/scala/org/knora/webapi/it/v2/KnoraSipiAuthenticationITSpec.scala b/integration/src/test/scala/org/knora/webapi/it/v2/KnoraSipiAuthenticationITSpec.scala index eeab4fd1ed..a35e330dd7 100644 --- a/integration/src/test/scala/org/knora/webapi/it/v2/KnoraSipiAuthenticationITSpec.scala +++ b/integration/src/test/scala/org/knora/webapi/it/v2/KnoraSipiAuthenticationITSpec.scala @@ -10,7 +10,6 @@ import org.apache.pekko.http.scaladsl.model.headers.BasicHttpCredentials import org.apache.pekko.http.scaladsl.unmarshalling.Unmarshal import zio.ZIO -import java.nio.file.Files import java.nio.file.Paths import scala.concurrent.Await import scala.concurrent.duration.* @@ -36,7 +35,6 @@ class KnoraSipiAuthenticationITSpec private val password = SharedTestDataADM.testPass private val marblesOriginalFilename = "marbles.tif" - private val pathToMarbles = Paths.get("..", s"test_data/test_route/images/$marblesOriginalFilename") override lazy val rdfDataObjects: List[RdfDataObject] = List( RdfDataObject( @@ -77,47 +75,6 @@ class KnoraSipiAuthenticationITSpec assert(response.status === StatusCodes.OK) } - "accept a token in Sipi that has been signed by Knora" in { - - // The image to be uploaded. - assert(Files.exists(pathToMarbles), s"File $pathToMarbles does not exist") - - // A multipart/form-data request containing the image. - val sipiFormData = Multipart.FormData( - Multipart.FormData.BodyPart( - "file", - HttpEntity.fromPath(MediaTypes.`image/tiff`, pathToMarbles), - Map("filename" -> pathToMarbles.getFileName.toString), - ), - ) - - // Send a POST request to Sipi, asking it to convert the image to JPEG 2000 and store it in a temporary file. - val sipiRequest = Post(s"$baseInternalSipiUrl/upload?token=$loginToken", sipiFormData) - val sipiResponse = singleAwaitingRequest(sipiRequest) - assert(sipiResponse.status == StatusCodes.OK) - } - - "not accept a token in Sipi that hasn't been signed by Knora" in { - val invalidToken = "a_invalid_token" - - // The image to be uploaded. - assert(Files.exists(pathToMarbles), s"File $pathToMarbles does not exist") - - // A multipart/form-data request containing the image. - val sipiFormData = Multipart.FormData( - Multipart.FormData.BodyPart( - "file", - HttpEntity.fromPath(MediaTypes.`image/tiff`, pathToMarbles), - Map("filename" -> pathToMarbles.getFileName.toString), - ), - ) - - // Send a POST request to Sipi, asking it to convert the image to JPEG 2000 and store it in a temporary file. - val sipiRequest = Post(s"$baseInternalSipiUrl/upload?token=$invalidToken", sipiFormData) - val sipiResponse = singleAwaitingRequest(sipiRequest) - assert(sipiResponse.status == StatusCodes.Unauthorized) - } - "accept a request with valid credentials to clean_temp_dir route which requires basic auth" in { // set the environment variables val username = "clean_tmp_dir_user" diff --git a/integration/src/test/scala/org/knora/webapi/it/v2/KnoraSipiIntegrationV2ITSpec.scala b/integration/src/test/scala/org/knora/webapi/it/v2/KnoraSipiIntegrationV2ITSpec.scala index 679587562d..79fb430d57 100644 --- a/integration/src/test/scala/org/knora/webapi/it/v2/KnoraSipiIntegrationV2ITSpec.scala +++ b/integration/src/test/scala/org/knora/webapi/it/v2/KnoraSipiIntegrationV2ITSpec.scala @@ -64,7 +64,6 @@ class KnoraSipiIntegrationV2ITSpec private val jsonLdHttpEntity = HttpEntity(RdfMediaTypes.`application/ld+json`, _: String) private val addAuthorization = addCredentials(BasicHttpCredentials(anythingUserEmail, password)) - private val addAssetIngested = addHeader("X-Asset-Ingested", "true") private val encodeUTF8 = URLEncoder.encode(_: String, "UTF-8") private val marblesOriginalFilename = "marbles.tif" @@ -72,12 +71,6 @@ class KnoraSipiIntegrationV2ITSpec private val marblesWidth = 1419 private val marblesHeight = 1001 - private val pathToMarblesWithWrongExtension = - Paths.get("..", "test_data/test_route/images/marbles_with_wrong_extension.jpg") - - private val jp2OriginalFilename = "67352ccc-d1b0-11e1-89ae-279075081939.jp2" - private val pathToJp2 = Paths.get("..", s"test_data/test_route/images/$jp2OriginalFilename") - private val trp88OriginalFilename = "Trp88.tiff" private val pathToTrp88 = Paths.get("..", s"test_data/test_route/images/$trp88OriginalFilename") private val trp88Width = 499 @@ -353,7 +346,7 @@ class KnoraSipiIntegrationV2ITSpec "create a resource with a still image file" in { // Upload the image to Sipi. val sipiUploadResponse: SipiUploadResponse = - uploadToSipi( + uploadToIngest( loginToken = loginToken, filesToUpload = Seq(FileToUpload(path = pathToMarbles, mimeType = org.apache.http.entity.ContentType.IMAGE_TIFF)), @@ -390,94 +383,23 @@ class KnoraSipiIntegrationV2ITSpec assert(savedImage.height == marblesHeight) } - "create a resource with a still image file without processing" in { - // Upload the image to Sipi. - val sipiUploadResponse: SipiUploadWithoutProcessingResponse = - uploadWithoutProcessingToSipi( - loginToken = loginToken, - filesToUpload = Seq(FileToUpload(path = pathToJp2, mimeType = org.apache.http.entity.ContentType.IMAGE_JPEG)), - ) - - val uploadedFile: SipiUploadWithoutProcessingResponseEntry = sipiUploadResponse.uploadedFiles.head - uploadedFile.filename should ===(jp2OriginalFilename) - - // Create the resource in the API. - - val jsonLdEntity = UploadFileRequest - .make(fileType = FileType.StillImageFile(), internalFilename = uploadedFile.filename) - .toJsonLd(className = Some("ThingPicture"), ontologyName = "anything") - - val responseJsonDoc = requestJsonLDWithAuth(Post(s"$baseApiUrl/v2/resources", jsonLdHttpEntity(jsonLdEntity))) - stillImageResourceIri.set(UnsafeZioRun.runOrThrow(responseJsonDoc.body.getRequiredIdValueAsKnoraDataIri).toString) - - val resource = getResponseJsonLD(Get(s"$baseApiUrl/v2/resources/${encodeUTF8(stillImageResourceIri.get)}")) - assert( - UnsafeZioRun - .runOrThrow(resource.body.getRequiredTypeAsKnoraApiV2ComplexTypeIri) - .toString == "http://0.0.0.0:3333/ontology/0001/anything/v2#ThingPicture", - ) - - // Get the new file value from the resource. - - val savedValues: JsonLDArray = getValuesFromResource(resource, HasStillImageFileValue.toSmartIri) - val savedValueObj: JsonLDObject = assertingUnique(savedValues.value).asOpt[JsonLDObject].get - stillImageFileValueIri.set(UnsafeZioRun.runOrThrow(savedValueObj.getRequiredIdValueAsKnoraDataIri).toString) - - val savedImage = savedValueToSavedImage(savedValueObj) - assert(savedImage.internalFilename == uploadedFile.filename) - } - "create a resource with a still image file that has already been ingested" in { // Create the resource in the API. val jsonLdEntity = UploadFileRequest .make(fileType = FileType.StillImageFile(), internalFilename = "De6XyNL4H71-D9QxghOuOPJ.jp2") .toJsonLd(className = Some("ThingPicture"), ontologyName = "anything") val response = requestJsonLDWithAuth( - Post(s"$baseApiUrl/v2/resources", jsonLdHttpEntity(jsonLdEntity)) ~> addAssetIngested, + Post(s"$baseApiUrl/v2/resources", jsonLdHttpEntity(jsonLdEntity)), ) val resIri = UnsafeZioRun.runOrThrow(response.body.getRequiredIdValueAsKnoraDataIri).toString val getRequest = Get(s"$baseApiUrl/v2/resources/${encodeUTF8(resIri)}") checkResponseOK(getRequest) } - "not create a resource with a still image file that has already been ingested if the header is not provided" in { - // Create the resource in the API. - val jsonLdEntity = UploadFileRequest - .make(fileType = FileType.StillImageFile(), internalFilename = "De6XyNL4H71-D9QxghOuOPJ.jp2") - .toJsonLd(className = Some("ThingPicture"), ontologyName = "anything") - // no X-Asset-Ingested header - val request = Post(s"$baseApiUrl/v2/resources", jsonLdHttpEntity(jsonLdEntity)) ~> addAuthorization - - val response = singleAwaitingRequest(request) - assert(response.status == StatusCodes.BadRequest) - val body = Await.result(Unmarshal(response.entity).to[String], 1.seconds) - assert( - body.contains( - "Asset 'De6XyNL4H71-D9QxghOuOPJ.jp2' not found in Sipi temp, when ingested with dsp-ingest you want to add the 'X-Asset-Ingested' header.", - ), - ) - } - - "reject an image file with the wrong file extension" in { - val exception = intercept[BadRequestException] { - uploadToSipi( - loginToken = loginToken, - filesToUpload = Seq( - FileToUpload( - path = pathToMarblesWithWrongExtension, - mimeType = org.apache.http.entity.ContentType.IMAGE_TIFF, - ), - ), - ) - } - - assert(exception.getMessage.contains("MIME type and/or file extension are inconsistent")) - } - "change a still image file value" in { // Upload the image to Sipi. val sipiUploadResponse: SipiUploadResponse = - uploadToSipi( + uploadToIngest( loginToken = loginToken, filesToUpload = Seq(FileToUpload(path = pathToTrp88, mimeType = org.apache.http.entity.ContentType.IMAGE_TIFF)), @@ -517,7 +439,7 @@ class KnoraSipiIntegrationV2ITSpec "create a resource with a PDF file" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq( FileToUpload(path = pathToMinimalPdf, mimeType = org.apache.http.entity.ContentType.create("application/pdf")), @@ -558,7 +480,7 @@ class KnoraSipiIntegrationV2ITSpec "change a PDF file value" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq( FileToUpload(path = pathToTestPdf, mimeType = org.apache.http.entity.ContentType.create("application/pdf")), @@ -594,39 +516,9 @@ class KnoraSipiIntegrationV2ITSpec checkResponseOK(sipiGetFileRequest) } - "change a PDF file value with X-Asset-Ingested=true" in { - // Update the value. - val jsonLdEntity = ChangeFileRequest - .make( - fileType = FileType.DocumentFile(), - internalFilename = "De6XyNL4H71-D9QxghOuOPJ.jp2", - resourceIri = pdfResourceIri.get, - valueIri = pdfValueIri.get, - className = Some("ThingDocument"), - ontologyName = "anything", - ) - .toJsonLd - - val response: JsonLDDocument = requestJsonLDWithAuth( - Put(s"$baseApiUrl/v2/values", jsonLdHttpEntity(jsonLdEntity)) ~> addAssetIngested, - ) - - val resource = getResponseJsonLD(Get(s"$baseApiUrl/v2/resources/${encodeUTF8(pdfResourceIri.get)}")) - - // Get the new file value from the resource. - val savedDocument: SavedDocument = savedValueToSavedDocument( - getValueFromResource( - resource = resource, - propertyIriInResult = OntologyConstants.KnoraApiV2Complex.HasDocumentFileValue.toSmartIri, - expectedValueIri = UnsafeZioRun.runOrThrow(response.body.getRequiredIdValueAsKnoraDataIri).toString, - ), - ) - assert(savedDocument.internalFilename == "De6XyNL4H71-D9QxghOuOPJ.jp2") - } - "not create a document resource if the file is actually a zip file" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq( FileToUpload(path = pathToMinimalZip, mimeType = org.apache.http.entity.ContentType.create("application/zip")), @@ -635,7 +527,6 @@ class KnoraSipiIntegrationV2ITSpec val uploadedFile: SipiUploadResponseEntry = sipiUploadResponse.uploadedFiles.head uploadedFile.originalFilename should ===(minimalZipOriginalFilename) - uploadedFile.fileType should equal("archive") // Create the resource in the API. @@ -649,7 +540,7 @@ class KnoraSipiIntegrationV2ITSpec "create a resource with a CSV file" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq(FileToUpload(path = pathToCsv1, mimeType = org.apache.http.entity.ContentType.create("text/csv"))), @@ -684,7 +575,7 @@ class KnoraSipiIntegrationV2ITSpec "change a CSV file value" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq(FileToUpload(path = pathToCsv2, mimeType = org.apache.http.entity.ContentType.create("text/csv"))), @@ -720,7 +611,7 @@ class KnoraSipiIntegrationV2ITSpec "not create a resource with a still image file that's actually a text file" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq(FileToUpload(path = pathToCsv1, mimeType = org.apache.http.entity.ContentType.create("text/csv"))), @@ -742,7 +633,7 @@ class KnoraSipiIntegrationV2ITSpec "create a resource with an XML file" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq(FileToUpload(path = pathToXml1, mimeType = org.apache.http.entity.ContentType.TEXT_XML)), ) @@ -782,7 +673,7 @@ class KnoraSipiIntegrationV2ITSpec "change an XML file value" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq(FileToUpload(path = pathToXml2, mimeType = org.apache.http.entity.ContentType.TEXT_XML)), ) @@ -823,7 +714,7 @@ class KnoraSipiIntegrationV2ITSpec "not create a resource of type TextRepresentation with a Zip file" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq( FileToUpload(path = pathToMinimalZip, mimeType = org.apache.http.entity.ContentType.create("application/zip")), @@ -846,7 +737,7 @@ class KnoraSipiIntegrationV2ITSpec "create a resource of type ArchiveRepresentation with a Zip file" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq( FileToUpload(path = pathToMinimalZip, mimeType = org.apache.http.entity.ContentType.create("application/zip")), @@ -891,7 +782,7 @@ class KnoraSipiIntegrationV2ITSpec "change a Zip file value" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq( FileToUpload(path = pathToTestZip, mimeType = org.apache.http.entity.ContentType.create("application/zip")), @@ -934,7 +825,7 @@ class KnoraSipiIntegrationV2ITSpec "create a resource of type ArchiveRepresentation with a 7z file" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq( FileToUpload( @@ -982,7 +873,7 @@ class KnoraSipiIntegrationV2ITSpec "create a resource with a WAV file" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq(FileToUpload(path = pathToMinimalWav, mimeType = org.apache.http.entity.ContentType.create("audio/wav"))), @@ -1026,7 +917,7 @@ class KnoraSipiIntegrationV2ITSpec "change a WAV file value" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq(FileToUpload(path = pathToTestWav, mimeType = org.apache.http.entity.ContentType.create("audio/wav"))), @@ -1068,7 +959,7 @@ class KnoraSipiIntegrationV2ITSpec "create a resource with a video file" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( loginToken = loginToken, filesToUpload = Seq(FileToUpload(path = pathToTestVideo, mimeType = org.apache.http.entity.ContentType.create("video/mp4"))), @@ -1112,10 +1003,14 @@ class KnoraSipiIntegrationV2ITSpec "change a video file value" in { // Upload the file to Sipi. - val sipiUploadResponse: SipiUploadResponse = uploadToSipi( - loginToken = loginToken, - filesToUpload = - Seq(FileToUpload(path = pathToTestVideo2, mimeType = org.apache.http.entity.ContentType.create("video/mp4"))), + val sipiUploadResponse: SipiUploadResponse = uploadToIngest( + loginToken, + Seq( + FileToUpload( + path = pathToTestVideo2, + mimeType = org.apache.http.entity.ContentType.create("video/mp4"), + ), + ), ) val uploadedFile: SipiUploadResponseEntry = sipiUploadResponse.uploadedFiles.head diff --git a/integration/src/test/scala/org/knora/webapi/it/v2/StandoffRouteV2ITSpec.scala b/integration/src/test/scala/org/knora/webapi/it/v2/StandoffRouteV2ITSpec.scala index ceb7e2c497..72b3de1649 100644 --- a/integration/src/test/scala/org/knora/webapi/it/v2/StandoffRouteV2ITSpec.scala +++ b/integration/src/test/scala/org/knora/webapi/it/v2/StandoffRouteV2ITSpec.scala @@ -37,6 +37,7 @@ import org.knora.webapi.models.filemodels.UploadFileRequest import org.knora.webapi.models.standoffmodels.DefineStandoffMapping import org.knora.webapi.sharedtestdata.SharedTestDataADM import org.knora.webapi.sharedtestdata.SharedTestDataADM2.anythingProjectIri +import org.knora.webapi.testservices.FileToUpload import org.knora.webapi.util.FileUtil import org.knora.webapi.util.MutableTestIri @@ -92,10 +93,11 @@ class StandoffRouteV2ITSpec extends ITKnoraLiveSpec with AuthenticationV2JsonPro HttpEntity.fromPath(MediaTypes.`text/xml`.toContentType(HttpCharsets.`UTF-8`), mappingFile), ), ) - val mappingRequest = Post(baseApiUrl + "/v2/mapping", formDataMapping) ~> addCredentials( - BasicHttpCredentials(anythingUserEmail, password), + singleAwaitingRequest( + Post(baseApiUrl + "/v2/mapping", formDataMapping) ~> addCredentials( + BasicHttpCredentials(anythingUserEmail, password), + ), ) - singleAwaitingRequest(mappingRequest) } def createResourceWithTextValue(xmlContent: String, mappingIRI: String): HttpResponse = { @@ -306,20 +308,15 @@ class StandoffRouteV2ITSpec extends ITKnoraLiveSpec with AuthenticationV2JsonPro assert(loginResponse.status == StatusCodes.OK, responseToString(loginResponse)) val loginToken = Await.result(Unmarshal(loginResponse.entity).to[LoginResponse], 1.seconds).token - // upload XSLT file to SIPI - val sipiFormData = Multipart.FormData( - Multipart.FormData.BodyPart( - "file", - HttpEntity.fromPath(ContentTypes.`text/xml(UTF-8)`, Paths.get(pathToFreetestXSLTFile)), - Map("filename" -> freetestXSLTFile), + val uploadedFile: SipiUploadResponseEntry = uploadToIngest( + loginToken = loginToken, + filesToUpload = Seq( + FileToUpload( + path = Paths.get(pathToFreetestXSLTFile), + mimeType = org.apache.http.entity.ContentType.create("text/xml"), + ), ), - ) - val sipiRequest = Post(s"${baseInternalSipiUrl}/upload?token=$loginToken", sipiFormData) - val sipiResponse = singleAwaitingRequest(sipiRequest) - val uploadedFile = { - import zio.json.* - responseToString(sipiResponse).fromJson[SipiUploadResponse].toOption.get.uploadedFiles.head - } + ).uploadedFiles.head // create FileRepresentation in API val uploadFileJson = UploadFileRequest @@ -328,9 +325,7 @@ class StandoffRouteV2ITSpec extends ITKnoraLiveSpec with AuthenticationV2JsonPro internalFilename = uploadedFile.internalFilename, resourceIRI = Some(freetestXSLTIRI), ) - .toJsonLd( - className = Some("XSLTransformation"), - ) + .toJsonLd(className = Some("XSLTransformation")) val fileRepresentationRequest = Post( s"$baseApiUrl/v2/resources", HttpEntity(RdfMediaTypes.`application/ld+json`, uploadFileJson), diff --git a/integration/src/test/scala/org/knora/webapi/responders/v2/ValuesResponderV2Spec.scala b/integration/src/test/scala/org/knora/webapi/responders/v2/ValuesResponderV2Spec.scala index 555c976c28..76c5cac3bf 100644 --- a/integration/src/test/scala/org/knora/webapi/responders/v2/ValuesResponderV2Spec.scala +++ b/integration/src/test/scala/org/knora/webapi/responders/v2/ValuesResponderV2Spec.scala @@ -32,10 +32,8 @@ import org.knora.webapi.messages.v2.responder.valuemessages.* import org.knora.webapi.models.filemodels.FileModelUtil import org.knora.webapi.models.filemodels.FileType import org.knora.webapi.routing.UnsafeZioRun -import org.knora.webapi.routing.v2.AssetIngestState import org.knora.webapi.sharedtestdata.SharedTestDataADM import org.knora.webapi.slice.admin.domain.model.User -import org.knora.webapi.store.iiif.errors.SipiException import org.knora.webapi.store.triplestore.api.TriplestoreService import org.knora.webapi.store.triplestore.api.TriplestoreService.Queries.Select import org.knora.webapi.util.MutableTestIri @@ -461,7 +459,7 @@ class ValuesResponderV2Spec extends CoreSpec with ImplicitSender { val propertyIri = "http://0.0.0.0:3333/ontology/0001/anything/v2#hasInteger".toSmartIri val intVal = IntegerValueContentV2(ApiV2Complex, 4) val duplicateValue = - CreateValueV2(resourceIri, resourceClassIri, propertyIri, intVal, ingestState = AssetIngestState.AssetInTemp) + CreateValueV2(resourceIri, resourceClassIri, propertyIri, intVal) val actual = UnsafeZioRun.run( ZIO.serviceWithZIO[ValuesResponderV2](_.createValueV2(duplicateValue, anythingUser1, randomUUID)), @@ -4412,80 +4410,6 @@ class ValuesResponderV2Spec extends CoreSpec with ImplicitSender { } - "not return a Sipi error if Sipi fails to delete a temporary file when Knora rejects a request" in { - val resourceIri: IRI = aThingPictureIri - val propertyIri: SmartIri = OntologyConstants.KnoraApiV2Complex.HasStillImageFileValue.toSmartIri - - val valueContent = StillImageFileValueContentV2( - ontologySchema = ApiV2Complex, - fileValue = FileValueV2( - internalFilename = "failure.jp2", // tells the mock Sipi responder to simulate failure - internalMimeType = mimeTypeJP2, - originalFilename = Some("test.tiff"), - originalMimeType = Some(mimeTypeTIFF), - None, - None, - ), - dimX = 512, - dimY = 256, - ) - - // Knora will reject this request. - val actual = UnsafeZioRun.run( - ZIO.serviceWithZIO[ValuesResponderV2]( - _.updateValueV2( - UpdateValueContentV2( - resourceIri = resourceIri, - resourceClassIri = thingPictureClassIri.toSmartIri, - propertyIri = propertyIri, - valueIri = stillImageFileValueIri.get, - valueContent = valueContent, - ), - incunabulaUser, // this user doesn't have the necessary permission - randomUUID, - ), - ), - ) - assertFailsWithA[ForbiddenException](actual) - } - - "return a Sipi error if Sipi fails to move a file to permanent storage" in { - val resourceIri: IRI = aThingPictureIri - val propertyIri: SmartIri = OntologyConstants.KnoraApiV2Complex.HasStillImageFileValue.toSmartIri - - val valueContent = StillImageFileValueContentV2( - ontologySchema = ApiV2Complex, - fileValue = FileValueV2( - internalFilename = "failure.jp2", // tells the mock Sipi responder to simulate failure - internalMimeType = mimeTypeJP2, - originalFilename = Some("test.tiff"), - originalMimeType = Some(mimeTypeTIFF), - None, - None, - ), - dimX = 512, - dimY = 256, - ) - - // Knora will accept this request, but the mock Sipi responder will say it failed to move the file to permanent storage. - val actual = UnsafeZioRun.run( - ZIO.serviceWithZIO[ValuesResponderV2]( - _.updateValueV2( - UpdateValueContentV2( - resourceIri = resourceIri, - resourceClassIri = thingPictureClassIri.toSmartIri, - propertyIri = propertyIri, - valueIri = stillImageFileValueIri.get, - valueContent = valueContent, - ), - anythingUser1, - randomUUID, - ), - ), - ) - assertFailsWithA[SipiException](actual) - } - "not delete a value if the requesting user does not have DeletePermission on the value" in { val resourceIri: IRI = aThingIri val propertyIri: SmartIri = "http://0.0.0.0:3333/ontology/0001/anything/v2#hasInteger".toSmartIri diff --git a/integration/src/test/scala/org/knora/webapi/testservices/TestClientService.scala b/integration/src/test/scala/org/knora/webapi/testservices/TestClientService.scala index fe7facf87d..9c69c39ad7 100644 --- a/integration/src/test/scala/org/knora/webapi/testservices/TestClientService.scala +++ b/integration/src/test/scala/org/knora/webapi/testservices/TestClientService.scala @@ -17,6 +17,7 @@ import sttp.client3.httpclient.zio.HttpClientZioBackend import zio.* import zio.json.* import zio.json.ast.Json +import zio.nio.file.Files import java.nio.file.Path import java.util.concurrent.TimeUnit @@ -29,9 +30,8 @@ import dsp.errors.BadRequestException import dsp.errors.NotFoundException import org.knora.webapi.config.AppConfig import org.knora.webapi.messages.store.sipimessages.SipiUploadResponse +import org.knora.webapi.messages.store.sipimessages.SipiUploadResponseEntry import org.knora.webapi.messages.store.sipimessages.SipiUploadResponseJsonProtocol.* -import org.knora.webapi.messages.store.sipimessages.SipiUploadWithoutProcessingResponse -import org.knora.webapi.messages.store.sipimessages.SipiUploadWithoutProcessingResponseJsonProtocol.* import org.knora.webapi.messages.store.triplestoremessages.TriplestoreJsonProtocol import org.knora.webapi.messages.util.rdf.JsonLDDocument import org.knora.webapi.messages.util.rdf.JsonLDUtil @@ -47,7 +47,11 @@ import pekko.http.scaladsl.unmarshalling.Unmarshal * @param path the path of the file. * @param mimeType the MIME type of the file. */ -final case class FileToUpload(path: Path, mimeType: ContentType) +final case class FileToUpload( + path: Path, + mimeType: ContentType, + shortcode: String = "0001", +) /** * Represents an image file to be uploaded to the IIF Service. @@ -65,7 +69,7 @@ final case class TestClientService( extends TriplestoreJsonProtocol with RequestBuilding { - private val targetHostUri = uri"${config.sipi.internalBaseUrl}" + private val ingestUrl = uri"${config.dspIngest.baseUrl}" implicit val executionContext: ExecutionContext = system.dispatchers.lookup(KnoraDispatchers.KnoraBlockingDispatcher) @@ -155,52 +159,30 @@ final case class TestClientService( } yield json /** - * Uploads a file to the IIIF Service's "upload" route and returns the information in Sipi's response. - * The upload creates a multipart/form-data request which can contain multiple files. + * Uploads a file to the Ingest service's "/projects/$shortcode/assets/ingest/$filename" route. * * @param loginToken the login token to be included in the request to Sipi. * @param files the files to be uploaded. * @return a [[SipiUploadResponse]] representing Sipi's response. */ - def uploadToSipi(loginToken: String, files: Seq[FileToUpload]): Task[SipiUploadResponse] = - for { - url <- ZIO.succeed(targetHostUri.addPath("upload")) - multiparts = files.map { file => - multipartFile("file", file.path.toFile) - .fileName(file.path.getFileName.toString) - .contentType(file.mimeType.toString) - } - response <- doSipiRequest( - quickRequest - .post(url) - .header("Authorization", s"Bearer $loginToken") - .multipartBody(multiparts), - ) - json <- ZIO.fromEither(response.fromJson[SipiUploadResponse]).mapError(Throwable(_)) - } yield json - - /** - * Uploads a file to the IIIF Service's "upload_without_processing" route and returns the information in Sipi's response. - * The upload creates a multipart/form-data request which can contain multiple files. - * - * @param loginToken the login token to be included in the request to Sipi. - * @param files the files to be uploaded. - * @return a [[SipiUploadWithoutProcessingResponse]] representing Sipi's response. - */ - def uploadWithoutProcessingToSipi( - loginToken: String, - files: Seq[FileToUpload], - ): Task[SipiUploadWithoutProcessingResponse] = - for { - url <- ZIO.succeed(targetHostUri.addPath("upload_without_processing").addParam("token", loginToken)) - multiparts = files.map { file => - multipartFile("file", file.path.toFile) - .fileName(file.path.getFileName.toString) - .contentType(file.mimeType.toString) - } - response <- doSipiRequest(quickRequest.post(url).multipartBody(multiparts)) - json <- ZIO.fromEither(response.fromJson[SipiUploadWithoutProcessingResponse]).mapError(Throwable(_)) - } yield json + def uploadToIngest(loginToken: String, filesToUpload: Seq[FileToUpload]): Task[SipiUploadResponse] = + ZIO + .foreach(filesToUpload) { file => + for { + contents <- Files.readAllBytes(zio.nio.file.Path.apply(file.path.toUri())) + url = ingestUrl.addPath("projects", file.shortcode, "assets", "ingest", file.path.getFileName.toString) + response <- + doSipiRequest( + quickRequest + .post(url) + .header("Content-Type", file.mimeType.toString) + .header("Authorization", s"Bearer $loginToken") + .body(contents.toArray), + ) + json <- ZIO.fromEither(response.fromJson[SipiUploadResponseEntry]).mapError(Throwable(_)) + } yield json + } + .map(responses => SipiUploadResponse(responses.toList)) private def doSipiRequest[T](request: Request[String, Any]): Task[String] = sttp.send(request).flatMap { response => diff --git a/sipi/config/sipi.docker-config.lua b/sipi/config/sipi.docker-config.lua index 4761452764..8a54962fcb 100644 --- a/sipi/config/sipi.docker-config.lua +++ b/sipi/config/sipi.docker-config.lua @@ -181,26 +181,11 @@ fileserver = { -- Custom routes. Each route is an URL path associated with a Lua script. -- routes = { - { - method = 'POST', - route = '/upload', - script = 'upload.lua' - }, - { - method = 'POST', - route = '/store', - script = 'store.lua' - }, { method = 'DELETE', route = '/delete_temp_file', script = 'delete_temp_file.lua' }, - { - method = 'POST', - route = '/upload_without_processing', - script = 'upload_without_processing.lua' - }, { method = 'GET', route = '/clean_temp_dir', diff --git a/sipi/config/sipi.docker-no-auth-config.lua b/sipi/config/sipi.docker-no-auth-config.lua index d651cef289..70e4f11ef2 100644 --- a/sipi/config/sipi.docker-no-auth-config.lua +++ b/sipi/config/sipi.docker-no-auth-config.lua @@ -190,16 +190,6 @@ routes = { route = '/test_knora_session_cookie', script = 'test_knora_session_cookie.lua' }, - { - method = 'POST', - route = '/upload', - script = 'upload.lua' - }, - { - method = 'POST', - route = '/store', - script = 'store.lua' - }, { method = 'DELETE', route = '/delete_temp_file', diff --git a/sipi/config/sipi.docker-test-config.lua b/sipi/config/sipi.docker-test-config.lua index 560b01ca01..84e9fcdbae 100644 --- a/sipi/config/sipi.docker-test-config.lua +++ b/sipi/config/sipi.docker-test-config.lua @@ -185,16 +185,6 @@ routes = { route = '/admin_upload', script = 'admin_upload.lua' }, - { - method = 'POST', - route = '/upload', - script = 'upload.lua' - }, - { - method = 'POST', - route = '/store', - script = 'store.lua' - }, { method = 'DELETE', route = '/delete_temp_file', diff --git a/sipi/config/sipi.local-config.lua b/sipi/config/sipi.local-config.lua index 02af8c24d9..be4e0feb83 100644 --- a/sipi/config/sipi.local-config.lua +++ b/sipi/config/sipi.local-config.lua @@ -148,16 +148,6 @@ fileserver = { -- executes the given script defined below -- routes = { - { - method = 'POST', - route = '/upload', - script = 'upload.lua' - }, - { - method = 'POST', - route = '/store', - script = 'store.lua' - }, { method = 'DELETE', route = '/delete_temp_file', diff --git a/sipi/scripts/delete_temp_file.lua b/sipi/scripts/delete_temp_file.lua deleted file mode 100644 index 28f17050da..0000000000 --- a/sipi/scripts/delete_temp_file.lua +++ /dev/null @@ -1,122 +0,0 @@ --- * Copyright © 2021 - 2024 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. --- * SPDX-License-Identifier: Apache-2.0 - --- --- Deletes a file from temporary storage. --- - -require "send_response" -require "authentication" - --- Buffer the response (helps with error handling). - -local success, error_msg = server.setBuffer() - -if not success then - send_error(500, "server.setBuffer() failed: " .. error_msg) - return -end - --- Check that this request is really from Knora and that the user has permission --- to delete the file. - -local token = auth_get_jwt_decoded() - -if token == nil then - return -end - -local knora_data = token["knora-data"] - -if knora_data == nil then - send_error(403, "No knora-data in token") - return -end - -if knora_data["permission"] ~= "DeleteTempFile" then - send_error(403, "Token does not grant permission to delete tmp file") - return -end - -local token_filename = knora_data["filename"] - -if token_filename == nil then - send_error(401, "Token does not specify a filename") - return -end - --- Parse the URL to get the filename to be deleted. - -local last_slash_pos = server.uri:match(".*/()") - -if last_slash_pos == nil then - send_error(400, "Invalid path: " .. server.uri) - return -end - -local filename = server.uri:sub(last_slash_pos) - -if filename:len() == 0 or filename == "." or filename == ".." then - send_error(400, "Invalid filename: " .. filename) - return -end - --- Check that the filename matches the one in the token. - -if filename ~= token_filename then - send_error(401, "Incorrect filename in token") - return -end - --- Check that the file exists in the temp directory. - -local hashed_filename -success, hashed_filename = helper.filename_hash(filename) - -if not success then - send_error(500, "helper.filename_hash() failed: " .. hashed_filename) - return -end - -local temp_file_path = config.imgroot .. '/tmp/' .. hashed_filename - -local exists -success, exists = server.fs.exists(temp_file_path) -if not success then - send_error(500, "server.fs.exists() failed: " .. exists) - return -end - -if not exists then - send_error(404, filename .. " not found") - return -end - -local filetype -success, filetype = server.fs.ftype(temp_file_path) -if not success then - send_error(500, "server.fs.ftype() failed: " .. filetype) - return -end - -if filetype ~= "FILE" then - send_error(400, filename .. " is not a file") - return -end - --- Delete the file. - -local errmsg -success, errmsg = server.fs.unlink(temp_file_path) -if not success then - send_error(500, "server.fs.unlink() failed: " .. errmsg) - return -end - -server.log("delete_temp_file.lua: deleted " .. temp_file_path, server.loglevel.LOG_DEBUG) - -local result = { - status = 0 -} - -send_success(result) diff --git a/sipi/scripts/store.lua b/sipi/scripts/store.lua deleted file mode 100644 index 70b59d4383..0000000000 --- a/sipi/scripts/store.lua +++ /dev/null @@ -1,212 +0,0 @@ --- * Copyright © 2021 - 2024 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. --- * SPDX-License-Identifier: Apache-2.0 --- --- Moves a file from temporary to permanent storage. --- - -require "file_specific_folder_util" -require "send_response" -require "authentication" - - --------------------------------------------------------------------------------- --- Get the basename of a string determined by removing the extension --------------------------------------------------------------------------------- -local function get_file_basename(path) - local temp = "" - local result = "" - local found = false - - for i = path:len(), 1, -1 do - if path:sub(i, i) ~= "." then - if found then - temp = temp .. path:sub(i, i) - end - else - found = true - end - end - - if found then - -- Reverse order of full file name - for j = temp:len(), 1, -1 do - result = result .. temp:sub(j, j) - end - else - result = path - end - - return result -end - - --- Buffer the response (helps with error handling). -local success, error_msg = server.setBuffer() -if not success then - send_error(500, "store.lua: server.setBuffer() failed: " .. error_msg) - return -end - --- Check for a valid JSON Web Token and permissions. -local token = auth_get_jwt_decoded() -if token == nil then - return -end -local knora_data = token["knora-data"] -if knora_data == nil then - send_error(403, "store.lua: No knora-data in token") - return -end -if knora_data["permission"] ~= "StoreFile" then - send_error(403, "store.lua: Token does not grant permission to store file") - return -end - --- get token filename -local token_filename = knora_data["filename"] -if token_filename == nil then - send_error(401, "store.lua: Token does not specify a filename") - return -end - --- get token prefix -local token_prefix = knora_data["prefix"] -if token_prefix == nil then - send_error(401, "store.lua: Token does not specify a prefix") - return -end -local prefix = server.post["prefix"] - -if prefix ~= token_prefix then - send_error(401, "store.lua: Incorrect prefix in token") - return -end - --- --- Get the submitted filename and check consistency. --- -if server.post == nil then - send_error(400, PARAMETERS_INCORRECT) - return -end -local filename = server.post["filename"] -if filename == nil then - send_error(400, PARAMETERS_INCORRECT) - return -end -if filename ~= token_filename then - send_error(401, "store.lua: Incorrect filename in token") - return -end - -server.log("store.lua: start processing " .. tostring(filename)) - --- --- Construct the path of that file under the temp directory. --- -local hashed_filename -success, hashed_filename = helper.filename_hash(filename) -if not success then - send_error(500, "store.lua: helper.filename_hash() failed: " .. hashed_filename) - return -end - -local tmp_folder_root = config.imgroot .. '/tmp' -local source_file = get_file_specific_path(tmp_folder_root, hashed_filename) -local source_preview = source_file:match("(.+)%..+") - --- --- Make sure the source file is readable. --- -local readable -success, readable = server.fs.is_readable(source_file) -if not success then - send_error(500, "store.lua: server.fs.is_readable() failed: " .. readable) - return -end -if not readable then - send_error(400, "store.lua: " .. source_file .. " not readable") - return -end - --- --- Move the temporary files to the permanent storage directory. --- -local project_folder = config.imgroot .. "/" .. prefix -success, error_msg = check_create_dir(project_folder) -if not success then - send_error(500, error_msg) - return -end -local destination_folder = check_and_create_file_specific_folder(project_folder, hashed_filename) -local destination_file = get_file_specific_path(project_folder, hashed_filename) -local destination_preview = destination_file:match("(.+)%..+") -success, error_msg = server.fs.moveFile(source_file, destination_file) -if not success then - send_error(500, - "store.lua: server.fs.moveFile() from " .. - tostring(source_file) .. " to " .. tostring(destination_file) .. " failed: " .. error_msg) - return -end - --- In case of a movie file, move the folder with the preview file to the permanent storage directory -local source_preview_exists -_, source_preview_exists = server.fs.exists(source_preview) -if source_preview_exists then - success, error_msg = os.rename(source_preview, destination_preview) - if not success then - send_error(500, "store.lua: moving folder with preview failed: " .. error_msg) - return - end -end - --- --- Move sidecar and original file to final storage location --- -local hashed_sidecar = get_file_basename(hashed_filename) .. ".info" -local source_sidecar = get_file_specific_path(tmp_folder_root, hashed_sidecar) -success, readable = server.fs.is_readable(source_sidecar) -if not success then - send_error(500, "store.lua: server.fs.is_readable() failed: " .. readable) - return -end - -if readable then - -- read the sidecar file into a string - local f = io.open(source_sidecar) - local jsonstr = f:read("*a") - f:close() - local sidecar - success, sidecar = server.json_to_table(jsonstr) - if not success then - send_error(500, "store.lua: server.json_to_table() failed: " .. sidecar) - return - end - - -- move sidecar file to storage location - local destination_sidecar = destination_folder .. "/" .. hashed_sidecar - success, error_msg = server.fs.moveFile(source_sidecar, destination_sidecar) - if not success then - send_error(500, "store.lua: server.fs.moveFile() from " .. - tostring(source_sidecar) .. " to " .. tostring(destination_sidecar) .. " failed: " .. error_msg) - return - end - - -- move the original file to the storage location - local source_original = get_file_specific_path(tmp_folder_root, sidecar["originalInternalFilename"]) - local destination_original = destination_folder .. "/" .. sidecar["originalInternalFilename"] - success, error_msg = server.fs.moveFile(source_original, destination_original) - if not success then - send_error(500, "store.lua: server.fs.moveFile() from " .. - tostring(source_original) .. " to " .. tostring(destination_original) .. " failed: " .. error_msg) - return - end - - server.log("store.lua: moved file " .. source_file .. " to " .. destination_file, server.loglevel.LOG_DEBUG) -end - -local result = { - status = 0 -} - -send_success(result) diff --git a/sipi/scripts/upload.lua b/sipi/scripts/upload.lua deleted file mode 100644 index 80b6f7537d..0000000000 --- a/sipi/scripts/upload.lua +++ /dev/null @@ -1,324 +0,0 @@ --- * Copyright © 2021 - 2024 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. --- * SPDX-License-Identifier: Apache-2.0 - --- --- Upload route for binary files. --- - -require "file_info" -require "send_response" -require "authentication" -require "util" -require "file_specific_folder_util" -local json = require "json" - - --- Buffer the response (helps with error handling). -local success, error_msg = server.setBuffer() -if not success then - send_error(500, "upload.lua: server.setBuffer() failed: " .. error_msg) - return -end - --- Check for a valid JSON Web Token. -local token = auth_get_jwt_decoded() -if token == nil then - return -end - --- Check that the temp folder is created -local tmp_folder_root = config.imgroot .. '/tmp' -success, error_msg = check_create_dir(tmp_folder_root) -if not success then - send_error(500, error_msg) - return -end - --- A table of data about each file that was uploaded. -local file_upload_data = {} - --- Process the uploaded files. -for file_index, file_params in pairs(server.uploads) do - -- - -- Check that the file's MIME type is supported. - -- - local mime_info - success, mime_info = server.file_mimetype(file_index) - if not success then - send_error(415, "upload.lua: server.file_mimetype() failed: " .. tostring(mime_info)) - return - end - local mime_type = mime_info["mimetype"] - if mime_type == nil then - send_error(415, "upload.lua: Could not determine MIME type of uploaded file") - return - end - - -- - -- get some more MIME type related information - -- - local original_filename = file_params["origname"] - local file_info = get_file_info(original_filename, mime_type) - - if file_info == nil then - server.log("upload.lua: file_info appears to be nil for: " .. tostring(original_filename), - server.loglevel.LOG_ERR) - send_error(415, "upload.lua: Unsupported MIME type: " .. tostring(mime_type)) - return - end - - -- Make a random filename for the temporary file. - local uuid62 - success, uuid62 = server.uuid62() - if not success then - send_error(500, "upload.lua: server.uuid62() failed: " .. uuid62) - return - end - - -- Construct response data about the file that was uploaded. - local media_type = file_info["media_type"] - - -- Add a subdirectory path if necessary. - local tmp_storage_filename - if media_type == IMAGE then - tmp_storage_filename = uuid62 .. ".jp2" - else - tmp_storage_filename = uuid62 .. "." .. file_info["extension"] - end - local hashed_tmp_storage_filename - success, hashed_tmp_storage_filename = helper.filename_hash(tmp_storage_filename) - if not success then - send_error(500, "upload.lua: helper.filename_hash() failed: " .. tostring(hashed_tmp_storage_filename)) - return - end - - -- filename for sidecar file - local tmp_storage_sidecar = uuid62 .. ".info" - local hashed_tmp_storage_sidecar - success, hashed_tmp_storage_sidecar = helper.filename_hash(tmp_storage_sidecar) - if not success then - send_error(500, "upload.lua: helper.filename_hash() failed: " .. tostring(hashed_tmp_storage_sidecar)) - return - end - - -- filename for original file copy - local tmp_storage_original = uuid62 .. "." .. file_info["extension"] .. ".orig" - local hashed_tmp_storage_original - success, hashed_tmp_storage_original = helper.filename_hash(tmp_storage_original) - if not success then - send_error(500, "upload.lua: helper.filename_hash() failed: " .. tostring(hashed_tmp_storage_original)) - return - end - - -- create tmp folder and subfolders for the files - local tmp_folder = check_and_create_file_specific_folder(tmp_folder_root, hashed_tmp_storage_filename) - - local tmp_storage_file_path = tmp_folder .. '/' .. hashed_tmp_storage_filename - local tmp_storage_sidecar_path = tmp_folder .. '/' .. hashed_tmp_storage_sidecar - local tmp_storage_original_path = tmp_folder .. '/' .. hashed_tmp_storage_original - - -- Create a IIIF base URL for the converted file. - local tmp_storage_url = get_external_protocol() .. "://" .. get_external_hostname() .. ":" .. get_external_port() .. - '/tmp/' .. hashed_tmp_storage_filename - - -- Copy original file also to tmp - success, error_msg = server.copyTmpfile(file_index, tmp_storage_original_path) - if not success then - send_error(500, "upload.lua: server.copyTmpfile() failed for " .. tostring(tmp_storage_original_path) .. ": " .. - tostring(error_msg)) - return - end - - -- Is this an image file? - if media_type == IMAGE then - -- - -- Yes. Create a new Lua image object. This reads the image into an - -- internal in-memory representation independent of the original image format. - -- - local uploaded_image - success, uploaded_image = SipiImage.new(file_index, { - original = original_filename, - hash = "sha256" - }) - if not success then - send_error(500, "upload.lua: SipiImage.new() failed: " .. tostring(uploaded_image)) - return - end - - -- Check that the file extension is correct for the file's MIME type. - local check - success, check = uploaded_image:mimetype_consistency(mime_type, original_filename) - if not success then - send_error(500, "upload.lua: uploaded_image:mimetype_consistency() failed: " .. check) - return - end - if not check then - send_error(400, MIMETYPES_INCONSISTENCY) - return - end - - -- Normalize image orientation to top-left -- - success, error_msg = uploaded_image:topleft() - if not success then - server.log( - "upload.lua: normalize image orientation failed for: " .. tostring(tmp_storage_file_path) .. ": " .. - tostring(error_msg), server.loglevel.LOG_ERR) - send_error(500, - "upload.lua: normalize image orientation failed for: " .. tostring(tmp_storage_file_path) .. ": " .. - tostring(error_msg)) - return - end - - -- Convert the image to JPEG 2000 format. - success, error_msg = uploaded_image:write(tmp_storage_file_path) - if not success then - send_error(500, - "upload.lua: uploaded_image:write() failed for " .. tostring(tmp_storage_file_path) .. ": " .. - tostring(error_msg)) - return - end - server.log("upload.lua: wrote image file to " .. tmp_storage_file_path, server.loglevel.LOG_DEBUG) - - -- Is this a video file? - elseif media_type == VIDEO then - success, error_msg = server.copyTmpfile(file_index, tmp_storage_file_path) - if not success then - send_error(500, "upload.lua: server.copyTmpfile() failed for " .. tostring(tmp_storage_file_path) .. ": " .. - tostring(error_msg)) - return - end - -- extract the frames from video file; they will be used for preview - local success_key_frames, error_msg_key_frames = os.execute("./scripts/export-moving-image-frames.sh -i " .. - tmp_storage_file_path) - if not success_key_frames then - send_error(500, "upload.lua: export-moving-image-frames.sh failed: " .. error_msg_key_frames) - return - end - server.log("upload.lua: wrote video file to " .. tmp_storage_file_path, server.loglevel.LOG_DEBUG) - else - -- It's neither an image nor a video file. Move it to its temporary storage location. - success, error_msg = server.copyTmpfile(file_index, tmp_storage_file_path) - if not success then - send_error(500, "upload.lua: server.copyTmpfile() failed for " .. tostring(tmp_storage_file_path) .. ": " .. - tostring(error_msg)) - return - end - server.log("upload.lua: wrote non-image file to " .. tmp_storage_file_path, server.loglevel.LOG_DEBUG) - end - - -- - -- Calculate checksum of original file - -- - local checksum_original = file_checksum(tmp_storage_original_path) - - -- - -- Calculate checksum of derivative file - -- - local checksum_derivative = file_checksum(tmp_storage_file_path) - - -- - -- prepare and write sidecar file - -- - local sidecar_data = {} - - if media_type == VIDEO then - local handle - local file_meta - -- get video file information with ffprobe: width, height, duration and frame rate (fps) - handle = io.popen( - "ffprobe -v error -select_streams v:0 -show_entries stream=width,height,bit_rate,duration,nb_frames,r_frame_rate -print_format json -i " .. - tmp_storage_file_path) - if handle ~= nil then - file_meta = handle:read("*a") - handle:close() - else - send_error(500, "upload.lua: running ffprobe failed for: " .. tostring(tmp_storage_file_path)) - return - end - - -- decode ffprobe output into json, but only first stream - local file_meta_json = json.decode(file_meta)['streams'][1] - - -- get video duration - local duration = tonumber(file_meta_json['duration']) - if not duration then - send_error(417, "upload.lua: ffprobe get duration failed: " .. duration) - end - -- get video width (dimX) - local width = tonumber(file_meta_json['width']); - if not width then - send_error(417, "upload.lua: ffprobe get width failed: " .. width) - end - -- get video height (dimY) - local height = tonumber(file_meta_json['height']) - if not height then - send_error(417, "upload.lua: ffprobe get height failed: " .. height) - end - -- get video fps - -- this is a bit tricky, because ffprobe returns something like 30/1 or 179/6; so, we have to convert into a floating point number; - -- or we can calculate fps from number of frames divided by duration - local fps - local frames = tonumber(file_meta_json['nb_frames']) - if not frames then - send_error(417, "upload.lua: ffprobe get frames failed: " .. frames) - else - fps = frames / duration - if not fps then - send_error(417, "upload.lua: ffprobe get fps failed: " .. fps) - end - end - - sidecar_data = { - originalFilename = original_filename, - checksumOriginal = checksum_original, - originalInternalFilename = hashed_tmp_storage_original, - internalFilename = tmp_storage_filename, - checksumDerivative = checksum_derivative, - width = width, - height = height, - duration = duration, - fps = fps - } - - -- TODO: similar setup for audio files; get duration with ffprobe and write extended sidecar data (DEV-770) - else - sidecar_data = { - originalFilename = original_filename, - checksumOriginal = checksum_original, - originalInternalFilename = hashed_tmp_storage_original, - internalFilename = tmp_storage_filename, - checksumDerivative = checksum_derivative - } - end - - local jsonstr - success, jsonstr = server.table_to_json(sidecar_data) - if not success then - send_error(500, "upload.lua: Couldn't create json string!") - return - end - local sidecar = io.open(tmp_storage_sidecar_path, "w") - if sidecar ~= nil then - sidecar:write(jsonstr) - sidecar:close() - else - send_error(500, "upload.lua: io.open() failed for " .. tostring(tmp_storage_sidecar_path)) - return - end - - local this_file_upload_data = { - internalFilename = tmp_storage_filename, - originalFilename = original_filename, - temporaryUrl = tmp_storage_url, - fileType = media_type, - sidecarFile = tmp_storage_sidecar, - checksumOriginal = checksum_original, - checksumDerivative = checksum_derivative - } - file_upload_data[file_index] = this_file_upload_data -end - --- Return the file upload data in the response. -local response = {} -response["uploadedFiles"] = file_upload_data -send_success(response) diff --git a/sipi/scripts/upload_without_processing.lua b/sipi/scripts/upload_without_processing.lua deleted file mode 100644 index 2203b2fb41..0000000000 --- a/sipi/scripts/upload_without_processing.lua +++ /dev/null @@ -1,73 +0,0 @@ --- Copyright © 2021 - 2024 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. --- SPDX-License-Identifier: Apache-2.0 --- - --- Upload route for binary files that skips transcoding. Directly puts the --- files in the temp folder. --- - -require "file_specific_folder_util" -require "authentication" -require "send_response" -require "util" - - --- Buffer the response (helps with error handling). -local success, error_msg -success, error_msg = server.setBuffer() -if not success then - send_error(500, "server.setBuffer() failed: " .. error_msg) - return -end - --- Check for a valid JSON Web Token from Knora. -local token = auth_get_jwt_decoded() -if token == nil then - return -end - --- Check that the root tmp folder is created -local tmp_folder_root = config.imgroot .. '/' .. 'tmp' -success, error_msg = check_create_dir(tmp_folder_root) -if not success then - send_error(500, error_msg) - return -end - --- A table of data about each file that was uploaded. -local file_upload_data = {} - --- Process the uploaded files. -for file_index, file_params in pairs(server.uploads) do - -- get the filename - local filename = file_params["origname"] - - -- create the file specific tmp folder - local tmp_folder = check_and_create_file_specific_folder(tmp_folder_root, filename) - - -- get the filepath - local tmp_storage_file_path = get_path_from_folder_and_filename(tmp_folder, filename) - - success, error_msg = server.copyTmpfile(file_index, tmp_storage_file_path) - if not success then - send_error(500, - "server.copyTmpfile() failed for " .. tostring(tmp_storage_file_path) .. ": " .. tostring(error_msg)) - return - end - server.log("upload_without_processing.lua: wrote file to " .. tmp_storage_file_path, server.loglevel.LOG_DEBUG) - - -- Calculate the checksum of the file - local checksum = file_checksum(tmp_storage_file_path) - - local this_file_upload_data = { - filename = filename, - checksum = checksum, - algorithm = "sha256" - } - file_upload_data[file_index] = this_file_upload_data -end - --- Return the file upload data in the response. -local response = {} -response["uploadedFiles"] = file_upload_data -send_success(response) diff --git a/webapi/src/main/scala/org/knora/webapi/messages/store/sipimessages/SipiMessages.scala b/webapi/src/main/scala/org/knora/webapi/messages/store/sipimessages/SipiMessages.scala index 0fc3eb2123..148581394e 100644 --- a/webapi/src/main/scala/org/knora/webapi/messages/store/sipimessages/SipiMessages.scala +++ b/webapi/src/main/scala/org/knora/webapi/messages/store/sipimessages/SipiMessages.scala @@ -17,26 +17,9 @@ import org.knora.webapi.slice.admin.domain.model.User */ sealed trait IIIFRequest extends StoreRequest with RelayedMessage +// NOTE: consider renaming all with Ingest prefixes sealed trait SipiRequest extends IIIFRequest -/** - * Asks Sipi to move a file from temporary to permanent storage. - * - * @param internalFilename the name of the file. - * @param prefix the prefix under which the file should be stored. - * @param requestingUser the user making the request. - */ -case class MoveTemporaryFileToPermanentStorageRequest(internalFilename: String, prefix: String, requestingUser: User) - extends SipiRequest - -/** - * Asks Sipi to delete a temporary file. - * - * @param internalFilename the name of the file. - * @param requestingUser the user making the request. - */ -case class DeleteTemporaryFileRequest(internalFilename: String, requestingUser: User) extends SipiRequest - /** * Asks Sipi for a text file. Currently only for UTF8 encoded text files. * @@ -59,27 +42,10 @@ case class SipiGetTextFileResponse(content: String) * * @param originalFilename the original filename that was submitted to Sipi. * @param internalFilename Sipi's internal filename for the stored temporary file. - * @param temporaryUrl the URL at which the temporary file can be accessed. - * @param fileType `image`, `text`, or `document`. */ case class SipiUploadResponseEntry( originalFilename: String, internalFilename: String, - temporaryUrl: String, - fileType: String, -) - -/** - * Represents the information that Sipi returns about each file that has been uploaded to the upload_without_processing route. - * - * @param filename the filename that was submitted to Sipi. - * @param checksum the checksum of the file - * @param algorithm the algorithm that was used to create the checksum of the file - */ -case class SipiUploadWithoutProcessingResponseEntry( - filename: String, - checksum: String, - algorithm: String, ) /** @@ -93,17 +59,3 @@ object SipiUploadResponseJsonProtocol { implicit val entry: JsonCodec[SipiUploadResponseEntry] = DeriveJsonCodec.gen[SipiUploadResponseEntry] implicit val response: JsonCodec[SipiUploadResponse] = DeriveJsonCodec.gen[SipiUploadResponse] } - -/** - * Represents Sipi's response to a file upload without processing request. - * - * @param uploadedFiles the information about each file that was uploaded. - */ -case class SipiUploadWithoutProcessingResponse(uploadedFiles: List[SipiUploadWithoutProcessingResponseEntry]) - -object SipiUploadWithoutProcessingResponseJsonProtocol { - implicit val withoutResponseEntryCodec: JsonCodec[SipiUploadWithoutProcessingResponseEntry] = - DeriveJsonCodec.gen[SipiUploadWithoutProcessingResponseEntry] - implicit val withoutResponseCodec: JsonCodec[SipiUploadWithoutProcessingResponse] = - DeriveJsonCodec.gen[SipiUploadWithoutProcessingResponse] -} diff --git a/webapi/src/main/scala/org/knora/webapi/messages/v2/responder/resourcemessages/ResourceMessagesV2.scala b/webapi/src/main/scala/org/knora/webapi/messages/v2/responder/resourcemessages/ResourceMessagesV2.scala index 43a2623aed..8f44105d97 100644 --- a/webapi/src/main/scala/org/knora/webapi/messages/v2/responder/resourcemessages/ResourceMessagesV2.scala +++ b/webapi/src/main/scala/org/knora/webapi/messages/v2/responder/resourcemessages/ResourceMessagesV2.scala @@ -26,8 +26,6 @@ import org.knora.webapi.messages.util.standoff.XMLUtil import org.knora.webapi.messages.v2.responder.* import org.knora.webapi.messages.v2.responder.standoffmessages.MappingXMLtoStandoff import org.knora.webapi.messages.v2.responder.valuemessages.* -import org.knora.webapi.routing.v2.AssetIngestState -import org.knora.webapi.routing.v2.AssetIngestState.* import org.knora.webapi.slice.admin.api.model.Project import org.knora.webapi.slice.admin.domain.model.KnoraProject.ProjectIri import org.knora.webapi.slice.admin.domain.model.Permission @@ -639,7 +637,6 @@ case class CreateResourceRequestV2( createResource: CreateResourceV2, requestingUser: User, apiRequestID: UUID, - ingestState: AssetIngestState = AssetInTemp, ) extends ResourcesResponderRequestV2 /** diff --git a/webapi/src/main/scala/org/knora/webapi/messages/v2/responder/valuemessages/ValueMessagesV2.scala b/webapi/src/main/scala/org/knora/webapi/messages/v2/responder/valuemessages/ValueMessagesV2.scala index 0cdb205ee9..88f0574fc7 100644 --- a/webapi/src/main/scala/org/knora/webapi/messages/v2/responder/valuemessages/ValueMessagesV2.scala +++ b/webapi/src/main/scala/org/knora/webapi/messages/v2/responder/valuemessages/ValueMessagesV2.scala @@ -40,8 +40,6 @@ import org.knora.webapi.messages.v2.responder.resourcemessages.ReadResourceV2 import org.knora.webapi.messages.v2.responder.standoffmessages.* import org.knora.webapi.messages.v2.responder.valuemessages.ValueContentV2.FileInfo import org.knora.webapi.routing.RouteUtilZ -import org.knora.webapi.routing.v2.AssetIngestState -import org.knora.webapi.routing.v2.AssetIngestState.* import org.knora.webapi.slice.admin.api.model.MaintenanceRequests.AssetId import org.knora.webapi.slice.admin.api.model.Project import org.knora.webapi.slice.admin.domain.model.KnoraProject.CopyrightAttribution @@ -479,7 +477,6 @@ case class ReadOtherValueV2( /** * Represents a Knora value to be created in an existing resource. * - * @param ingestState indicates the state of the file, either ingested or in temp folder * @param resourceIri the resource the new value should be attached to. * @param resourceClassIri the resource class that the client believes the resource belongs to. * @param propertyIri the property of the new value. If the client wants to create a link, this must be a link value property. @@ -499,7 +496,6 @@ case class CreateValueV2( valueUUID: Option[UUID] = None, valueCreationDate: Option[Instant] = None, permissions: Option[String] = None, - ingestState: AssetIngestState = AssetInTemp, ) /** A trait for classes representing information to be updated in a value. */ @@ -555,7 +551,6 @@ case class UpdateValueContentV2( permissions: Option[String] = None, valueCreationDate: Option[Instant] = None, newValueVersionIri: Option[SmartIri] = None, - ingestState: AssetIngestState = AssetInTemp, ) extends UpdateValueV2 /** @@ -681,7 +676,6 @@ object ValueContentV2 { * Given the jsonLd contains a FileValueHasFilename, it will try to fetch the FileInfo from Sipi or Dsp-Ingest. * * @param shortcode The shortcode of the project - * @param ingestState The state of the file, either ingested already or in Sipi temp folder * @param jsonLd the jsonLd object * @return Some FileInfo if FileValueHasFilename found and the remote service returned the metadata. * None if FileValueHasFilename is not found in the jsonLd object. @@ -689,51 +683,29 @@ object ValueContentV2 { */ def getFileInfo( shortcode: Shortcode, - ingestState: AssetIngestState, jsonLd: JsonLDObject, ): ZIO[SipiService, Throwable, Option[FileInfo]] = - val filenameMaybe = jsonLd.getString(FileValueHasFilename).toOption.flatten - fileInfoFromExternal(filenameMaybe, ingestState, shortcode) + fileInfoFromExternal(jsonLd.getString(FileValueHasFilename).toOption.flatten, shortcode) def fileInfoFromExternal( filenameMaybe: Option[String], - state: AssetIngestState, shortcode: Shortcode, ): ZIO[SipiService, Throwable, Option[FileInfo]] = - (filenameMaybe, state) match - case (None, _) => ZIO.none - case (Some(filename), AssetIngested) => fileInfoFromDspIngest(shortcode, filename).asSome - case (Some(filename), AssetInTemp) => fileInfoFromSipi(filename).asSome - - private def fileInfoFromSipi(filename: String) = - ZIO.serviceWithZIO[SipiService]( - _.getFileMetadataFromSipiTemp(filename) - .mapBoth( - { - case NotFoundException(_) => - NotFoundException( - s"Asset '$filename' not found in Sipi temp, when ingested with dsp-ingest you want to add the 'X-Asset-Ingested' header.", - ) - case e => e - }, - FileInfo(filename, _), - ), - ) - - private def fileInfoFromDspIngest(shortcode: Shortcode, filename: String) = - for { - sipiService <- ZIO.service[SipiService] - assetId <- ZIO - .fromEither(AssetId.from(filename.substring(0, filename.indexOf('.')))) - .mapError(msg => BadRequestException(s"Invalid value for 'fileValueHasFilename': $msg")) - meta <- sipiService.getFileMetadataFromDspIngest(shortcode, assetId).mapError { - case NotFoundException(_) => - NotFoundException( - s"Asset '$filename' not found in dsp-ingest, when ingested to Sipi temp you want to remove the 'X-Asset-Ingested' header.", - ) - case e => e - } - } yield FileInfo(filename, meta) + ZIO.foreach(filenameMaybe) { filename => + for { + sipiService <- ZIO.service[SipiService] + assetId <- ZIO + .fromEither(AssetId.from(filename.substring(0, filename.indexOf('.')))) + .mapError(msg => BadRequestException(s"Invalid value for 'fileValueHasFilename': $msg")) + meta <- sipiService.getFileMetadataFromDspIngest(shortcode, assetId).mapError { + case NotFoundException(_) => + NotFoundException( + s"Asset '$filename' not found in dsp-ingest, make sure the old Sipi upload mechanism is not being used.", + ) + case e => e + } + } yield FileInfo(filename, meta) + } } /** diff --git a/webapi/src/main/scala/org/knora/webapi/responders/v2/ResourceUtilV2.scala b/webapi/src/main/scala/org/knora/webapi/responders/v2/ResourceUtilV2.scala index 039d676977..3600279150 100644 --- a/webapi/src/main/scala/org/knora/webapi/responders/v2/ResourceUtilV2.scala +++ b/webapi/src/main/scala/org/knora/webapi/responders/v2/ResourceUtilV2.scala @@ -14,17 +14,10 @@ import org.knora.webapi.IRI import org.knora.webapi.messages.OntologyConstants import org.knora.webapi.messages.SmartIri import org.knora.webapi.messages.StringFormatter -import org.knora.webapi.messages.store.sipimessages.DeleteTemporaryFileRequest -import org.knora.webapi.messages.store.sipimessages.MoveTemporaryFileToPermanentStorageRequest import org.knora.webapi.messages.twirl.queries.sparql import org.knora.webapi.messages.util.PermissionUtilADM -import org.knora.webapi.messages.v2.responder.UpdateResultInProject import org.knora.webapi.messages.v2.responder.resourcemessages.ReadResourceV2 -import org.knora.webapi.messages.v2.responder.valuemessages.FileValueContentV2 import org.knora.webapi.messages.v2.responder.valuemessages.ReadValueV2 -import org.knora.webapi.messages.v2.responder.valuemessages.StillImageExternalFileValueContentV2 -import org.knora.webapi.routing.v2.AssetIngestState -import org.knora.webapi.routing.v2.AssetIngestState.* import org.knora.webapi.slice.admin.domain.model.Permission import org.knora.webapi.slice.admin.domain.model.User import org.knora.webapi.store.iiif.api.SipiService @@ -73,35 +66,6 @@ trait ResourceUtilV2 { * @return Task of Either None for nonexistent, true for root and false for child node. */ def checkListNodeExistsAndIsRootNode(nodeIri: IRI): Task[Either[Option[Nothing], Boolean]] - - /** - * Given an update task which changes [[FileValueContentV2]] values the related files need to be finalized. - * If the update was successful the temporary files are moved to permanent storage. - * If the update failed the temporary files are deleted silently. - * - * @param updateTask the [[Task]] that updates the triplestore. - * @param fileValues the values which the task updates. - * @param requestingUser the user making the request. - * - * @return The result of the updateTask, unless this task was successful and the subsequent move to permanent storage failed. - * In the latter case the failure from Sipi is returned. - */ - def doSipiPostUpdate[T <: UpdateResultInProject]( - updateTask: Task[T], - fileValues: Seq[FileValueContentV2], - requestingUser: User, - ): Task[T] - - def doSipiPostUpdateIfInTemp[T <: UpdateResultInProject]( - ingestState: AssetIngestState, - updateTask: Task[T], - fileValues: Seq[FileValueContentV2], - requestingUser: User, - ): Task[T] = - ingestState match { - case AssetIngested => updateTask - case AssetInTemp => doSipiPostUpdate(updateTask, fileValues, requestingUser) - } } final case class ResourceUtilV2Live(triplestore: TriplestoreService, sipiService: SipiService) @@ -195,48 +159,6 @@ final case class ResourceUtilV2Live(triplestore: TriplestoreService, sipiService } yield maybeList } - - /** - * Given a future representing an operation that was supposed to update a value in a triplestore, checks whether - * the updated value was a file value. If not, this method returns the same future. If it was a file value, this - * method checks whether the update was successful. If so, it asks Sipi to move the file to permanent storage. - * If not, it asks Sipi to delete the temporary file. - * - * @param updateTask the Task that should have updated the triplestore. - * @param valueContents: Seq[FileValueContentV2], the value that should have been created or updated. - * @param requestingUser the user making the request. - */ - override def doSipiPostUpdate[T <: UpdateResultInProject]( - updateTask: Task[T], - valueContents: Seq[FileValueContentV2], - requestingUser: User, - ): Task[T] = { - val temporaryFiles = valueContents.filterNot(_.is[StillImageExternalFileValueContentV2]) - updateTask.foldZIO( - (e: Throwable) => { - ZIO - .foreachDiscard(temporaryFiles) { file => - sipiService - .deleteTemporaryFile(DeleteTemporaryFileRequest(file.fileValue.internalFilename, requestingUser)) - .logError - } - .ignore *> ZIO.fail(e) - }, - (updateInProject: T) => { - ZIO - .foreachDiscard(temporaryFiles) { file => - sipiService.moveTemporaryFileToPermanentStorage( - MoveTemporaryFileToPermanentStorageRequest( - file.fileValue.internalFilename, - updateInProject.projectADM.shortcode, - requestingUser, - ), - ) - } - .as(updateInProject) - }, - ) - } } object ResourceUtilV2Live { diff --git a/webapi/src/main/scala/org/knora/webapi/responders/v2/StandoffResponderV2.scala b/webapi/src/main/scala/org/knora/webapi/responders/v2/StandoffResponderV2.scala index 31f5b15a2f..b8d80f4594 100644 --- a/webapi/src/main/scala/org/knora/webapi/responders/v2/StandoffResponderV2.scala +++ b/webapi/src/main/scala/org/knora/webapi/responders/v2/StandoffResponderV2.scala @@ -27,7 +27,6 @@ import org.knora.webapi.core.MessageRelay import org.knora.webapi.messages.* import org.knora.webapi.messages.IriConversions.* import org.knora.webapi.messages.store.sipimessages.SipiGetTextFileRequest -import org.knora.webapi.messages.store.sipimessages.SipiGetTextFileResponse import org.knora.webapi.messages.twirl.MappingElement import org.knora.webapi.messages.twirl.MappingStandoffDatatypeClass import org.knora.webapi.messages.twirl.MappingXMLAttribute @@ -52,6 +51,7 @@ import org.knora.webapi.slice.infrastructure.CacheManager import org.knora.webapi.slice.infrastructure.EhCache import org.knora.webapi.slice.ontology.domain.model.Cardinality.AtLeastOne import org.knora.webapi.slice.ontology.domain.model.Cardinality.ExactlyOne +import org.knora.webapi.store.iiif.impl.SipiServiceLive import org.knora.webapi.store.triplestore.api.TriplestoreService import org.knora.webapi.store.triplestore.api.TriplestoreService.Queries.Construct import org.knora.webapi.store.triplestore.api.TriplestoreService.Queries.Update @@ -70,11 +70,12 @@ final case class StandoffResponderV2( projectService: ProjectService, xsltCache: EhCache[String, String], mappingCache: EhCache[String, MappingXMLtoStandoff], + sipiServiceLive: SipiServiceLive, )(implicit val stringFormatter: StringFormatter) extends MessageHandler with LazyLogging { - private val xmlMimeTypes = Set("text/xml", "application/xml") + private val xmlMimeTypes = Set("text/xml", "application/xml", "application/xslt+xml") override def isResponsibleFor(message: ResponderRequest): Boolean = message.isInstanceOf[StandoffResponderRequestV2] @@ -181,8 +182,8 @@ final case class StandoffResponderV2( } else { for { response <- - messageRelay - .ask[SipiGetTextFileResponse]( + sipiServiceLive + .getTextFileRequest( SipiGetTextFileRequest( fileUrl = xsltFileUrl, requestingUser = KnoraSystemInstances.Users.SystemUser, @@ -979,7 +980,8 @@ object StandoffResponderV2 { xc <- ZIO.serviceWithZIO[CacheManager](_.createCache[String, String]("xsltCache")) mc <- ZIO.serviceWithZIO[CacheManager](_.createCache[String, MappingXMLtoStandoff]("mappingCache")) sf <- ZIO.service[StringFormatter] - handler <- mr.subscribe(StandoffResponderV2(ac, mr, ts, cru, stu, ps, xc, mc)(sf)) + ssl <- ZIO.service[SipiServiceLive] + handler <- mr.subscribe(StandoffResponderV2(ac, mr, ts, cru, stu, ps, xc, mc, ssl)(sf)) } yield handler } } diff --git a/webapi/src/main/scala/org/knora/webapi/responders/v2/ValuesResponderV2.scala b/webapi/src/main/scala/org/knora/webapi/responders/v2/ValuesResponderV2.scala index 2911e3955f..06b66d73a0 100644 --- a/webapi/src/main/scala/org/knora/webapi/responders/v2/ValuesResponderV2.scala +++ b/webapi/src/main/scala/org/knora/webapi/responders/v2/ValuesResponderV2.scala @@ -308,7 +308,7 @@ final case class ValuesResponderV2( ) } - val triplestoreUpdateFuture: Task[CreateValueResponseV2] = for { + for { // Don't allow anonymous users to create values. _ <- ZIO.when(requestingUser.isAnonymousUser)( ZIO.fail(ForbiddenException("Anonymous users aren't allowed to create values")), @@ -316,15 +316,6 @@ final case class ValuesResponderV2( // Do the remaining pre-update checks and the update while holding an update lock on the resource. taskResult <- IriLocker.runWithIriLock(apiRequestID, valueToCreate.resourceIri, taskZio) } yield taskResult - - // If we were creating a file value, have Sipi move the file to permanent storage if the update - // was successful, or delete the temporary file if the update failed. - resourceUtilV2.doSipiPostUpdateIfInTemp( - valueToCreate.ingestState, - triplestoreUpdateFuture, - valueToCreate.valueContent.asOpt[FileValueContentV2].toSeq, - requestingUser, - ) } private def ifIsListValueThenCheckItPointsToListNodeWhichIsNotARootNode(valueContent: ValueContentV2) = @@ -950,19 +941,12 @@ final case class ValuesResponderV2( updateValue match { case updateValueContentV2: UpdateValueContentV2 => // This is a request to update the content of a value. - val triplestoreUpdate = IriLocker.runWithIriLock( + IriLocker.runWithIriLock( apiRequestId, updateValueContentV2.resourceIri, makeTaskFutureToUpdateValueContent(updateValueContentV2), ) - resourceUtilV2.doSipiPostUpdateIfInTemp( - updateValueContentV2.ingestState, - triplestoreUpdate, - updateValueContentV2.valueContent.asOpt[FileValueContentV2].toSeq, - requestingUser, - ) - case updateValuePermissionsV2: UpdateValuePermissionsV2 => // This is a request to update the permissions attached to a value. IriLocker.runWithIriLock( diff --git a/webapi/src/main/scala/org/knora/webapi/responders/v2/resources/CreateResourceV2Handler.scala b/webapi/src/main/scala/org/knora/webapi/responders/v2/resources/CreateResourceV2Handler.scala index 5d9401c97e..0a42465cab 100644 --- a/webapi/src/main/scala/org/knora/webapi/responders/v2/resources/CreateResourceV2Handler.scala +++ b/webapi/src/main/scala/org/knora/webapi/responders/v2/resources/CreateResourceV2Handler.scala @@ -83,12 +83,7 @@ final case class CreateResourceV2Handler( * @return a [[ReadResourcesSequenceV2]] containing a preview of the resource. */ def apply(createResourceRequestV2: CreateResourceRequestV2): Task[ReadResourcesSequenceV2] = - resourceUtilV2.doSipiPostUpdateIfInTemp( - createResourceRequestV2.ingestState, - triplestoreUpdate(createResourceRequestV2), - createResourceRequestV2.createResource.flatValues.flatMap(_.valueContent.asOpt[FileValueContentV2]).toSeq, - createResourceRequestV2.requestingUser, - ) + triplestoreUpdate(createResourceRequestV2) private def triplestoreUpdate( createResourceRequestV2: CreateResourceRequestV2, diff --git a/webapi/src/main/scala/org/knora/webapi/routing/v2/AssetIngestedState.scala b/webapi/src/main/scala/org/knora/webapi/routing/v2/AssetIngestedState.scala deleted file mode 100644 index 9b81865ef6..0000000000 --- a/webapi/src/main/scala/org/knora/webapi/routing/v2/AssetIngestedState.scala +++ /dev/null @@ -1,19 +0,0 @@ -/* - * Copyright © 2021 - 2024 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. - * SPDX-License-Identifier: Apache-2.0 - */ - -package org.knora.webapi.routing.v2 - -import org.apache.pekko.http.scaladsl.model.HttpHeader - -sealed trait AssetIngestState - -object AssetIngestState { - case object AssetIngested extends AssetIngestState - case object AssetInTemp extends AssetIngestState - - def headerAssetIngestState(headers: Seq[HttpHeader]): AssetIngestState = - if (headers.exists(_.name == "X-Asset-Ingested")) AssetIngested - else AssetInTemp -} diff --git a/webapi/src/main/scala/org/knora/webapi/routing/v2/ResourcesRouteV2.scala b/webapi/src/main/scala/org/knora/webapi/routing/v2/ResourcesRouteV2.scala index 3ea0850490..c313b03815 100644 --- a/webapi/src/main/scala/org/knora/webapi/routing/v2/ResourcesRouteV2.scala +++ b/webapi/src/main/scala/org/knora/webapi/routing/v2/ResourcesRouteV2.scala @@ -104,9 +104,8 @@ final case class ResourcesRouteV2(appConfig: AppConfig)( val requestTask = for { requestingUser <- ZIO.serviceWithZIO[Authenticator](_.getUserADM(requestContext)) apiRequestId <- RouteUtilZ.randomUuid() - ingestState = AssetIngestState.headerAssetIngestState(requestContext.request.headers) requestMessage <- jsonLdRequestParser( - _.createResourceRequestV2(jsonRequest, ingestState, requestingUser, apiRequestId), + _.createResourceRequestV2(jsonRequest, requestingUser, apiRequestId), ).mapError(BadRequestException.apply) // check for each value which represents a file value if the file's MIME type is allowed _ <- checkMimeTypesForFileValueContents(requestMessage.createResource.flatValues) diff --git a/webapi/src/main/scala/org/knora/webapi/routing/v2/ValuesRouteV2.scala b/webapi/src/main/scala/org/knora/webapi/routing/v2/ValuesRouteV2.scala index 5944972226..60d3b8da5d 100644 --- a/webapi/src/main/scala/org/knora/webapi/routing/v2/ValuesRouteV2.scala +++ b/webapi/src/main/scala/org/knora/webapi/routing/v2/ValuesRouteV2.scala @@ -86,9 +86,8 @@ final case class ValuesRouteV2()( for { requestingUser <- ZIO.serviceWithZIO[Authenticator](_.getUserADM(ctx)) apiRequestId <- Random.nextUUID - ingestState = AssetIngestState.headerAssetIngestState(ctx.request.headers) valueToCreate <- jsonLdRequestParser( - _.createValueV2FromJsonLd(jsonLdString, ingestState).mapError(BadRequestException(_)), + _.createValueV2FromJsonLd(jsonLdString).mapError(BadRequestException(_)), ) response <- responder(_.createValueV2(valueToCreate, requestingUser, apiRequestId)) } yield response, @@ -107,9 +106,8 @@ final case class ValuesRouteV2()( for { requestingUser <- ZIO.serviceWithZIO[Authenticator](_.getUserADM(ctx)) apiRequestId <- Random.nextUUID - ingestState = AssetIngestState.headerAssetIngestState(ctx.request.headers) updateValue <- jsonLdRequestParser( - _.updateValueV2fromJsonLd(jsonLdString, ingestState).mapError(BadRequestException(_)), + _.updateValueV2fromJsonLd(jsonLdString).mapError(BadRequestException(_)), ) response <- responder(_.updateValueV2(updateValue, requestingUser, apiRequestId)) } yield response, diff --git a/webapi/src/main/scala/org/knora/webapi/slice/admin/api/PermissionsEndpoints.scala b/webapi/src/main/scala/org/knora/webapi/slice/admin/api/PermissionsEndpoints.scala index afe5eedd6c..494e032a52 100644 --- a/webapi/src/main/scala/org/knora/webapi/slice/admin/api/PermissionsEndpoints.scala +++ b/webapi/src/main/scala/org/knora/webapi/slice/admin/api/PermissionsEndpoints.scala @@ -42,7 +42,6 @@ object PermissionEndpointsRequests { forProperty.isDefined } object ChangeDoapRequest { - import org.knora.webapi.slice.admin.api.Codecs.ZioJsonCodec.projectIri given JsonCodec[ChangeDoapRequest] = DeriveJsonCodec.gen[ChangeDoapRequest] } } diff --git a/webapi/src/main/scala/org/knora/webapi/slice/common/ApiComplexV2JsonLdRequestParser.scala b/webapi/src/main/scala/org/knora/webapi/slice/common/ApiComplexV2JsonLdRequestParser.scala index e1aec3eb14..bca0bbe893 100644 --- a/webapi/src/main/scala/org/knora/webapi/slice/common/ApiComplexV2JsonLdRequestParser.scala +++ b/webapi/src/main/scala/org/knora/webapi/slice/common/ApiComplexV2JsonLdRequestParser.scala @@ -28,7 +28,6 @@ import org.knora.webapi.messages.v2.responder.resourcemessages.DeleteOrEraseReso import org.knora.webapi.messages.v2.responder.resourcemessages.UpdateResourceMetadataRequestV2 import org.knora.webapi.messages.v2.responder.valuemessages.* import org.knora.webapi.messages.v2.responder.valuemessages.ValueContentV2.FileInfo -import org.knora.webapi.routing.v2.AssetIngestState import org.knora.webapi.slice.admin.api.model.Project import org.knora.webapi.slice.admin.domain.model.KnoraProject.ProjectIri import org.knora.webapi.slice.admin.domain.model.KnoraProject.Shortcode @@ -230,7 +229,6 @@ final case class ApiComplexV2JsonLdRequestParser( def createResourceRequestV2( str: String, - ingestState: AssetIngestState, requestingUser: User, uuid: UUID, ): IO[String, CreateResourceRequestV2] = ZIO.scoped { @@ -244,7 +242,7 @@ final case class ApiComplexV2JsonLdRequestParser( .fail("Resource IRI and project IRI must reference the same project") .when(r.resourceIri.exists(_.shortcode != project.getShortcode)) attachedToUser <- attachedToUser(r.resource, requestingUser, project.projectIri) - values <- extractValues(r.resource, project.getShortcode, ingestState) + values <- extractValues(r.resource, project.getShortcode) } yield CreateResourceRequestV2( CreateResourceV2( r.resourceIri.map(_.smartIri), @@ -257,14 +255,12 @@ final case class ApiComplexV2JsonLdRequestParser( ), attachedToUser, uuid, - ingestState, ) } private def extractValues( r: Resource, shortcode: Shortcode, - ingestState: AssetIngestState, ): IO[String, Map[SmartIri, Seq[CreateValueInNewResourceV2]]] = val filteredProperties = Seq( RDF.`type`.toString, @@ -280,17 +276,16 @@ final case class ApiComplexV2JsonLdRequestParser( .filter(p => !filteredProperties.contains(p.getPredicate.toString)) .toSeq ZIO - .foreach(valueStatements)(valueStatementAsContent(_, shortcode, ingestState)) + .foreach(valueStatements)(valueStatementAsContent(_, shortcode)) .map(_.groupMap(_._1.smartIri)(_._2)) private def valueStatementAsContent( statement: Statement, shortcode: Shortcode, - ingestState: AssetIngestState, ): IO[String, (PropertyIri, CreateValueInNewResourceV2)] = for { v <- ValueResource.from(statement) - cnt <- getValueContent(v, shortcode, ingestState) + cnt <- getValueContent(v, shortcode) customValueUuid <- v.valueHasUuidOption customValueCreationDate <- v.valueCreationDateOption permissions <- v.hasPermissionsOption @@ -338,7 +333,7 @@ final case class ApiComplexV2JsonLdRequestParser( ) .unsome - def updateValueV2fromJsonLd(str: String, ingestState: AssetIngestState): IO[String, UpdateValueV2] = + def updateValueV2fromJsonLd(str: String): IO[String, UpdateValueV2] = ZIO.scoped { for { r <- RootResource.fromJsonLd(str) @@ -348,7 +343,7 @@ final case class ApiComplexV2JsonLdRequestParser( valueCreationDate <- v.valueCreationDateOption valuePermissions <- v.hasPermissionsOption newValueVersionIri <- newValueVersionIri(v, valueIri) - valueContent <- getValueContent(v, resourceIri.shortcode, ingestState).map(Some(_)).orElse(ZIO.none) + valueContent <- getValueContent(v, resourceIri.shortcode).map(Some(_)).orElse(ZIO.none) updateValue <- (valueContent, valuePermissions) match case (Some(valueContentV2), _) => ZIO.succeed( @@ -361,7 +356,6 @@ final case class ApiComplexV2JsonLdRequestParser( valuePermissions, valueCreationDate, newValueVersionIri.map(_.smartIri), - ingestState, ), ) case (_, Some(permissions)) => @@ -381,7 +375,7 @@ final case class ApiComplexV2JsonLdRequestParser( } yield updateValue } - def createValueV2FromJsonLd(str: String, ingestState: AssetIngestState): IO[String, CreateValueV2] = + def createValueV2FromJsonLd(str: String): IO[String, CreateValueV2] = ZIO.scoped { for { r <- RootResource.fromJsonLd(str) @@ -390,7 +384,7 @@ final case class ApiComplexV2JsonLdRequestParser( valueUuid <- v.valueHasUuidOption valueCreationDate <- v.valueCreationDateOption valuePermissions <- v.hasPermissionsOption - valueContent <- getValueContent(v, resourceIri.shortcode, ingestState) + valueContent <- getValueContent(v, resourceIri.shortcode) } yield CreateValueV2( resourceIri.smartIri.toString, r.resourceClassSmartIri, @@ -400,14 +394,12 @@ final case class ApiComplexV2JsonLdRequestParser( valueUuid, valueCreationDate, valuePermissions, - ingestState, ) } private def getValueContent( v: ValueResource, shortcode: Shortcode, - ingestState: AssetIngestState, ): IO[String, ValueContentV2] = def withFileInfo[T](fileInfo: Option[FileInfo], f: FileInfo => Either[String, T]): IO[String, T] = fileInfo match @@ -418,7 +410,7 @@ final case class ApiComplexV2JsonLdRequestParser( valueResource = v.r i <- ValueContentV2 - .fileInfoFromExternal(maybeFileName, ingestState, shortcode) + .fileInfoFromExternal(maybeFileName, shortcode) .provide(ZLayer.succeed(sipiService)) .mapError(_.getMessage) content <- diff --git a/webapi/src/main/scala/org/knora/webapi/store/iiif/IIIFRequestMessageHandler.scala b/webapi/src/main/scala/org/knora/webapi/store/iiif/IIIFRequestMessageHandler.scala index 3364ea90ca..dae3962ee1 100644 --- a/webapi/src/main/scala/org/knora/webapi/store/iiif/IIIFRequestMessageHandler.scala +++ b/webapi/src/main/scala/org/knora/webapi/store/iiif/IIIFRequestMessageHandler.scala @@ -10,9 +10,7 @@ import zio.* import org.knora.webapi.core.MessageHandler import org.knora.webapi.core.MessageRelay import org.knora.webapi.messages.ResponderRequest -import org.knora.webapi.messages.store.sipimessages.DeleteTemporaryFileRequest import org.knora.webapi.messages.store.sipimessages.IIIFRequest -import org.knora.webapi.messages.store.sipimessages.MoveTemporaryFileToPermanentStorageRequest import org.knora.webapi.messages.store.sipimessages.SipiGetTextFileRequest import org.knora.webapi.store.iiif.api.SipiService @@ -24,10 +22,8 @@ final case class IIIFRequestMessageHandlerLive(iiifService: SipiService) extends message.isInstanceOf[IIIFRequest] override def handle(message: ResponderRequest): Task[Any] = message match { - case req: MoveTemporaryFileToPermanentStorageRequest => iiifService.moveTemporaryFileToPermanentStorage(req) - case req: DeleteTemporaryFileRequest => iiifService.deleteTemporaryFile(req) - case req: SipiGetTextFileRequest => iiifService.getTextFileRequest(req) - case other => ZIO.logError(s"IIIFServiceManager received an unexpected message: $other") + case req: SipiGetTextFileRequest => iiifService.getTextFileRequest(req) + case other => ZIO.logError(s"IIIFServiceManager received an unexpected message: $other") } } diff --git a/webapi/src/main/scala/org/knora/webapi/store/iiif/api/SipiService.scala b/webapi/src/main/scala/org/knora/webapi/store/iiif/api/SipiService.scala index 96ae4ef350..59e8f535dd 100644 --- a/webapi/src/main/scala/org/knora/webapi/store/iiif/api/SipiService.scala +++ b/webapi/src/main/scala/org/knora/webapi/store/iiif/api/SipiService.scala @@ -11,7 +11,6 @@ import zio.json.JsonDecoder import zio.nio.file.Path import org.knora.webapi.messages.store.sipimessages.* -import org.knora.webapi.messages.v2.responder.SuccessResponseV2 import org.knora.webapi.slice.admin.api.model.MaintenanceRequests.AssetId import org.knora.webapi.slice.admin.domain.model.KnoraProject.Shortcode import org.knora.webapi.slice.admin.domain.model.User @@ -61,14 +60,6 @@ object FileMetadataSipiResponse { trait SipiService { - /** - * Asks Sipi for metadata about a file in the tmp folder, served from the 'knora.json' route. - * - * @param filename the path to the file. - * @return a [[FileMetadataSipiResponse]] containing the requested metadata. - */ - def getFileMetadataFromSipiTemp(filename: String): Task[FileMetadataSipiResponse] - /** * Asks DSP-Ingest for metadata about a file in permanent location, served from the 'knora.json' route. * @@ -78,24 +69,6 @@ trait SipiService { */ def getFileMetadataFromDspIngest(shortcode: Shortcode, assetId: AssetId): Task[FileMetadataSipiResponse] - /** - * Asks Sipi to move a file from temporary storage to permanent storage. - * - * @param moveTemporaryFileToPermanentStorageRequestV2 the request. - * @return a [[SuccessResponseV2]]. - */ - def moveTemporaryFileToPermanentStorage( - moveTemporaryFileToPermanentStorageRequestV2: MoveTemporaryFileToPermanentStorageRequest, - ): Task[SuccessResponseV2] - - /** - * Asks Sipi to delete a temporary file. - * - * @param deleteTemporaryFileRequestV2 the request. - * @return a [[SuccessResponseV2]]. - */ - def deleteTemporaryFile(deleteTemporaryFileRequestV2: DeleteTemporaryFileRequest): Task[SuccessResponseV2] - /** * Asks Sipi for a text file used internally by Knora. * @@ -112,5 +85,4 @@ trait SipiService { * @return The path to the downloaded asset. If the asset could not be downloaded, [[None]] is returned. */ def downloadAsset(asset: Asset, targetDir: Path, user: User): Task[Option[Path]] - } diff --git a/webapi/src/main/scala/org/knora/webapi/store/iiif/impl/SipiServiceLive.scala b/webapi/src/main/scala/org/knora/webapi/store/iiif/impl/SipiServiceLive.scala index 40339e1c73..1bf6e14850 100644 --- a/webapi/src/main/scala/org/knora/webapi/store/iiif/impl/SipiServiceLive.scala +++ b/webapi/src/main/scala/org/knora/webapi/store/iiif/impl/SipiServiceLive.scala @@ -10,10 +10,7 @@ import sttp.client3 import sttp.client3.* import sttp.client3.SttpBackend import sttp.client3.httpclient.zio.HttpClientZioBackend -import sttp.model.Uri import zio.* -import zio.json.DecoderOps -import zio.json.ast.Json import zio.nio.file.Path import java.net.URI @@ -22,8 +19,6 @@ import dsp.errors.BadRequestException import dsp.errors.NotFoundException import org.knora.webapi.config.Sipi import org.knora.webapi.messages.store.sipimessages.* -import org.knora.webapi.messages.util.KnoraSystemInstances -import org.knora.webapi.messages.v2.responder.SuccessResponseV2 import org.knora.webapi.slice.admin.api.model.MaintenanceRequests.AssetId import org.knora.webapi.slice.admin.domain.model.KnoraProject.Shortcode import org.knora.webapi.slice.admin.domain.model.User @@ -31,7 +26,6 @@ import org.knora.webapi.slice.admin.domain.service.Asset import org.knora.webapi.slice.admin.domain.service.DspIngestClient import org.knora.webapi.slice.infrastructure.Jwt import org.knora.webapi.slice.infrastructure.JwtService -import org.knora.webapi.slice.infrastructure.Scope as AuthScope import org.knora.webapi.slice.security.ScopeResolver import org.knora.webapi.store.iiif.api.FileMetadataSipiResponse import org.knora.webapi.store.iiif.api.SipiService @@ -54,24 +48,6 @@ final case class SipiServiceLive( s"${sipiConfig.internalBaseUrl}/${asset.belongsToProject.value}/${asset.internalFilename}" } - /** - * Asks Sipi for metadata about a file, served from the 'knora.json' route. - * - * @param filename the file name - * @return a [[FileMetadataSipiResponse]] containing the requested metadata. - */ - override def getFileMetadataFromSipiTemp(filename: String): Task[FileMetadataSipiResponse] = - for { - jwt <- jwtService.createJwt(KnoraSystemInstances.Users.SystemUser.userIri, AuthScope.admin) - request = quickRequest - .get(uri"${sipiConfig.internalBaseUrl}/tmp/$filename/knora.json") - .header("Authorization", s"Bearer ${jwt.jwtString}") - body <- doSipiRequest(request) - res <- ZIO - .fromEither(body.fromJson[FileMetadataSipiResponse]) - .mapError(e => SipiException(s"Invalid response from Sipi: $e, $body")) - } yield res - override def getFileMetadataFromDspIngest(shortcode: Shortcode, assetId: AssetId): Task[FileMetadataSipiResponse] = for { response <- dspIngestClient.getAssetInfo(shortcode, assetId) @@ -86,64 +62,6 @@ final case class SipiServiceLive( response.fps.map(BigDecimal(_)), ) - /** - * Asks Sipi to move a file from temporary storage to permanent storage. - * - * @param moveTemporaryFileToPermanentStorageRequestV2 the request. - * @return a [[SuccessResponseV2]]. - */ - def moveTemporaryFileToPermanentStorage( - moveTemporaryFileToPermanentStorageRequestV2: MoveTemporaryFileToPermanentStorageRequest, - ): Task[SuccessResponseV2] = { - val user = moveTemporaryFileToPermanentStorageRequestV2.requestingUser - val params = Map( - ("filename" -> moveTemporaryFileToPermanentStorageRequestV2.internalFilename), - ("prefix" -> moveTemporaryFileToPermanentStorageRequestV2.prefix), - ) - for { - scope <- scopeResolver.resolve(user) - token <- jwtService.createJwt( - user.userIri, - scope, - Map( - "knora-data" -> Json.Obj( - "permission" -> Json.Str("StoreFile"), - "filename" -> Json.Str(moveTemporaryFileToPermanentStorageRequestV2.internalFilename), - "prefix" -> Json.Str(moveTemporaryFileToPermanentStorageRequestV2.prefix), - ), - ), - ) - url = uri"${sipiConfig.internalBaseUrl}/${sipiConfig.moveFileRoute}?token=${token.jwtString}" - _ <- doSipiRequest(quickRequest.post(url).body(params)) - } yield SuccessResponseV2("Moved file to permanent storage.") - } - - /** - * Asks Sipi to delete a temporary file. - * - * @param deleteTemporaryFileRequestV2 the request. - * @return a [[SuccessResponseV2]]. - */ - def deleteTemporaryFile(deleteTemporaryFileRequestV2: DeleteTemporaryFileRequest): Task[SuccessResponseV2] = { - val deleteRequestContent = - Map( - "knora-data" -> Json.Obj( - "permission" -> Json.Str("DeleteTempFile"), - "filename" -> Json.Str(deleteTemporaryFileRequestV2.internalFilename), - ), - ) - - val url: String => Uri = s => - uri"${sipiConfig.internalBaseUrl}/${sipiConfig.deleteTempFileRoute}/${deleteTemporaryFileRequestV2.internalFilename}?token=${s}" - - val user = deleteTemporaryFileRequestV2.requestingUser - for { - scope <- scopeResolver.resolve(user) - token <- jwtService.createJwt(user.userIri, scope, deleteRequestContent) - _ <- doSipiRequest(quickRequest.delete(url(token.jwtString))) - } yield SuccessResponseV2("Deleted temporary file.") - } - /** * Asks Sipi for a text file used internally by Knora. * diff --git a/webapi/src/main/scala/org/knora/webapi/store/iiif/impl/SipiServiceMock.scala b/webapi/src/main/scala/org/knora/webapi/store/iiif/impl/SipiServiceMock.scala index d08916813a..1d52c16357 100644 --- a/webapi/src/main/scala/org/knora/webapi/store/iiif/impl/SipiServiceMock.scala +++ b/webapi/src/main/scala/org/knora/webapi/store/iiif/impl/SipiServiceMock.scala @@ -9,7 +9,6 @@ import zio.* import zio.nio.file.Path import org.knora.webapi.messages.store.sipimessages.* -import org.knora.webapi.messages.v2.responder.SuccessResponseV2 import org.knora.webapi.slice.admin.api.model.MaintenanceRequests.AssetId import org.knora.webapi.slice.admin.domain.model.KnoraProject import org.knora.webapi.slice.admin.domain.model.KnoraProject.Shortcode @@ -27,11 +26,6 @@ import org.knora.webapi.store.iiif.impl.SipiServiceMock.SipiMockMethodName.* */ case class SipiServiceMock(ref: Ref[Map[SipiMockMethodName, Task[Object]]]) extends SipiService { - /** - * A request with this filename will always cause a Sipi error. - */ - private val FAILURE_FILENAME: String = "failure.jp2" - private def getReturnValue[T](method: SipiMockMethodName): Task[T] = ref.get.flatMap( _.getOrElse( @@ -48,25 +42,6 @@ case class SipiServiceMock(ref: Ref[Map[SipiMockMethodName, Task[Object]]]) exte ref.set(SipiMockMethodName.values.map(_ -> fail).toMap) } - override def getFileMetadataFromSipiTemp(filename: String): Task[FileMetadataSipiResponse] = - getReturnValue(GetFileMetadataFromSipiTemp) - - def moveTemporaryFileToPermanentStorage( - moveTemporaryFileToPermanentStorageRequestV2: MoveTemporaryFileToPermanentStorageRequest, - ): Task[SuccessResponseV2] = - if (moveTemporaryFileToPermanentStorageRequestV2.internalFilename == FAILURE_FILENAME) { - ZIO.fail(SipiException("Sipi failed to move file to permanent storage")) - } else { - ZIO.succeed(SuccessResponseV2("Moved file to permanent storage")) - } - - def deleteTemporaryFile(deleteTemporaryFileRequestV2: DeleteTemporaryFileRequest): Task[SuccessResponseV2] = - if (deleteTemporaryFileRequestV2.internalFilename == FAILURE_FILENAME) { - ZIO.fail(SipiException("Sipi failed to delete temporary file")) - } else { - ZIO.succeed(SuccessResponseV2("Deleted temporary file")) - } - override def getTextFileRequest(textFileRequest: SipiGetTextFileRequest): Task[SipiGetTextFileResponse] = getReturnValue(GetTextFileRequest) @@ -99,7 +74,8 @@ object SipiServiceMock { Ref .make[Map[SipiMockMethodName, Task[Object]]]( Map( - GetFileMetadataFromSipiTemp -> ZIO.succeed(defaultGetFileMetadataFromSipiTempResponse), + GetFileMetadataFromSipiTemp -> ZIO.succeed(defaultGetFileMetadataFromSipiTempResponse), + GetFileMetadataFromDspIngest -> ZIO.succeed(defaultGetFileMetadataFromSipiTempResponse), ), ) .map(ref => SipiServiceMock(ref)), diff --git a/webapi/src/test/scala/org/knora/webapi/messages/v2/responder/valuemessages/ValueContentV2Spec.scala b/webapi/src/test/scala/org/knora/webapi/messages/v2/responder/valuemessages/ValueContentV2Spec.scala index ae976f68cd..c8bf7b3de5 100644 --- a/webapi/src/test/scala/org/knora/webapi/messages/v2/responder/valuemessages/ValueContentV2Spec.scala +++ b/webapi/src/test/scala/org/knora/webapi/messages/v2/responder/valuemessages/ValueContentV2Spec.scala @@ -9,21 +9,13 @@ import zio.Task import zio.ZIO import zio.ZLayer import zio.nio.file.Path -import zio.test.Assertion.failsWithA import zio.test.Spec import zio.test.ZIOSpecDefault -import zio.test.assert import zio.test.assertTrue -import dsp.errors.AssertionException -import org.knora.webapi.messages.store.sipimessages.DeleteTemporaryFileRequest -import org.knora.webapi.messages.store.sipimessages.MoveTemporaryFileToPermanentStorageRequest import org.knora.webapi.messages.store.sipimessages.SipiGetTextFileRequest import org.knora.webapi.messages.store.sipimessages.SipiGetTextFileResponse import org.knora.webapi.messages.util.rdf.JsonLDUtil -import org.knora.webapi.messages.v2.responder.SuccessResponseV2 -import org.knora.webapi.routing.v2.AssetIngestState -import org.knora.webapi.routing.v2.AssetIngestState.* import org.knora.webapi.slice.admin.api.model.MaintenanceRequests.AssetId import org.knora.webapi.slice.admin.domain.model.KnoraProject import org.knora.webapi.slice.admin.domain.model.User @@ -44,54 +36,23 @@ object ValueContentV2Spec extends ZIOSpecDefault { override def spec: Spec[Any, Option[Throwable]] = suite("ValueContentV2.getFileInfo")( - suite("Given the asset is present in the tmp folder of Sipi")( - test("When getting file metadata with AssetInTemp from Sipi, then it should succeed") { - for { - temp <- ValueContentV2.getFileInfo(shortcode0001, AssetInTemp, jsonLdObj).some - } yield assertTrue(temp.metadata == expected) - }, - test("When getting file metadata with AssetIngested from dsp-ingest, then it should fail") { - for { - exit <- ValueContentV2.getFileInfo(shortcode0001, AssetIngested, jsonLdObj).exit - } yield assert(exit)(failsWithA[AssertionException]) - }, - ).provide(mockSipi(AssetInTemp)), suite("Given the asset is ingested")( - test("When getting file metadata with AssetInTemp from Sipi, then it should fail") { - for { - exit <- ValueContentV2.getFileInfo(shortcode0001, AssetInTemp, jsonLdObj).exit - } yield assert(exit)(failsWithA[AssertionException]) - }, - test("When getting file metadata with AssetIngested from dsp-ingest, then it should succeed") { + test("When getting file metadata from dsp-ingest, then it should succeed") { for { - ingested <- ValueContentV2.getFileInfo(shortcode0001, AssetIngested, jsonLdObj).some + ingested <- ValueContentV2.getFileInfo(shortcode0001, jsonLdObj).some } yield assertTrue(ingested.metadata == expected) }, - ).provide(mockSipi(AssetIngested)), + ).provide(mockSipi()), ) - private def mockSipi(flag: AssetIngestState) = ZLayer.succeed(new SipiService { - - override def getFileMetadataFromSipiTemp(filename: String): Task[FileMetadataSipiResponse] = - if (flag == AssetInTemp) { ZIO.succeed(expected) } - else { ZIO.fail(AssertionException("fail")) } - + private def mockSipi() = ZLayer.succeed(new SipiService { override def getFileMetadataFromDspIngest( shortcode: KnoraProject.Shortcode, assetId: AssetId, ): Task[FileMetadataSipiResponse] = - if (flag == AssetIngested) { ZIO.succeed(expected) } - else { ZIO.fail(AssertionException("fail")) } + ZIO.succeed(expected) // The following are unsupported operations because they are not used in the test - def moveTemporaryFileToPermanentStorage( - moveTemporaryFileToPermanentStorageRequestV2: MoveTemporaryFileToPermanentStorageRequest, - ): Task[SuccessResponseV2] = - ZIO.dieMessage("unsupported operation") - def deleteTemporaryFile( - deleteTemporaryFileRequestV2: DeleteTemporaryFileRequest, - ): Task[SuccessResponseV2] = - ZIO.dieMessage("unsupported operation") def getTextFileRequest(textFileRequest: SipiGetTextFileRequest): Task[SipiGetTextFileResponse] = ZIO.dieMessage("unsupported operation") def downloadAsset(asset: Asset, targetDir: Path, user: User): Task[Option[Path]] = diff --git a/webapi/src/test/scala/org/knora/webapi/slice/common/ApiComplexV2JsonLdRequestParserSpec.scala b/webapi/src/test/scala/org/knora/webapi/slice/common/ApiComplexV2JsonLdRequestParserSpec.scala index 57dc9bf502..7715a6b827 100644 --- a/webapi/src/test/scala/org/knora/webapi/slice/common/ApiComplexV2JsonLdRequestParserSpec.scala +++ b/webapi/src/test/scala/org/knora/webapi/slice/common/ApiComplexV2JsonLdRequestParserSpec.scala @@ -43,7 +43,6 @@ import org.knora.webapi.messages.v2.responder.valuemessages.TextValueType.Unform import org.knora.webapi.messages.v2.responder.valuemessages.TimeValueContentV2 import org.knora.webapi.messages.v2.responder.valuemessages.UriValueContentV2 import org.knora.webapi.responders.IriService -import org.knora.webapi.routing.v2.AssetIngestState.AssetIngested import org.knora.webapi.slice.admin.domain.model.* import org.knora.webapi.slice.admin.domain.repo.* import org.knora.webapi.slice.admin.domain.service.* @@ -134,20 +133,20 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { test("getResourceIri should get the id") { check(Gen.fromIterable(Seq(createIntegerValue, createLinkValue).map(_.toJsonPretty))) { json => for { - actual <- service(_.createValueV2FromJsonLd(json, AssetIngested)) + actual <- service(_.createValueV2FromJsonLd(json)) } yield assertTrue(actual.resourceIri == "http://rdfh.ch/0001/a-thing") } }, test("rootResourceClassIri should get the rdfs:type") { check(Gen.fromIterable(Seq(createIntegerValue, createLinkValue).map(_.toJsonPretty))) { json => for { - actual <- service(_.createValueV2FromJsonLd(json, AssetIngested)) + actual <- service(_.createValueV2FromJsonLd(json)) } yield assertTrue(actual.resourceClassIri.toString == "http://0.0.0.0:3333/ontology/0001/anything/v2#Thing") } }, test("value property should be present") { for { - actual <- service(_.createValueV2FromJsonLd(createIntegerValue.toJsonPretty, AssetIngested)) + actual <- service(_.createValueV2FromJsonLd(createIntegerValue.toJsonPretty)) } yield assertTrue( actual.propertyIri == sf.toSmartIri("http://0.0.0.0:3333/ontology/0001/anything/v2#hasInteger"), ) @@ -172,7 +171,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue(actual.valueContent == IntegerValueContentV2(ApiV2Complex, 4, None)) @@ -197,7 +195,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue(actual.valueContent == DecimalValueContentV2(ApiV2Complex, BigDecimal(4), None)) @@ -222,7 +219,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue(actual.valueContent == BooleanValueContentV2(ApiV2Complex, true, None)) @@ -244,7 +240,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue(actual.valueContent == GeomValueContentV2(ApiV2Complex, "{}", None)) @@ -273,7 +268,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue( @@ -300,7 +294,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue( @@ -329,7 +322,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue( @@ -363,7 +355,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue( @@ -390,7 +381,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue(actual.valueContent == GeonameValueContentV2(ApiV2Complex, "foo", None)) @@ -415,7 +405,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue(actual.valueContent == ColorValueContentV2(ApiV2Complex, "red", None)) @@ -441,7 +430,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue( @@ -475,7 +463,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue( @@ -514,7 +501,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue( @@ -548,7 +534,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue( @@ -575,7 +560,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue( @@ -602,7 +586,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue( @@ -629,7 +612,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue( @@ -657,7 +639,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue( @@ -696,7 +677,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { | "xsd": "http://www.w3.org/2001/XMLSchema#" | } |}""".stripMargin, - AssetIngested, ), ) } yield assertTrue( @@ -726,7 +706,7 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { "http://api.knora.org/ontology/knora-api/v2#textValueHasLanguage":"en" } }""".stripMargin) - value <- service(_.createValueV2FromJsonLd(str, AssetIngested)) + value <- service(_.createValueV2FromJsonLd(str)) } yield assertTrue( value == CreateValueV2( resourceIri = "http://rdfh.ch/0001/a-thing", @@ -747,7 +727,6 @@ object ApiComplexV2JsonLdRequestParserSpec extends ZIOSpecDefault { valueUUID = None, valueCreationDate = None, permissions = None, - ingestState = AssetIngested, ), ) }