Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GridDyna] Add some batch CRUD endpoints #111

Merged
merged 24 commits into from
Jul 12, 2024
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
fdd3458
[GridDyna] Add some batch CRUD endpoints
thangqp May 7, 2024
6a29931
Set to current version of filter library
thangqp May 7, 2024
3680536
Update URI path
thangqp May 7, 2024
2549a85
Merge branch 'main' into dynamic_simulation_add_some_batch_crud_endpo…
thangqp May 15, 2024
db4c530
Merge branch 'main' into dynamic_simulation_add_some_batch_crud_endpo…
thangqp May 21, 2024
588ae16
Merge from main
thangqp Jun 5, 2024
dcebe97
Add tests for new endpoints
thangqp Jun 6, 2024
c508737
Sonar
thangqp Jun 6, 2024
4834285
merge from main
thangqp Jun 11, 2024
1190a4b
Add modify all endpoint
thangqp Jun 11, 2024
2251b0c
Add tests
thangqp Jun 11, 2024
c0ac2fc
Update to filter library 1.0.8
thangqp Jun 18, 2024
d508e0f
Merge branch 'main' into dynamic_simulation_add_some_batch_crud_endpo…
thangqp Jun 18, 2024
f551b4c
Update description for endpoints
thangqp Jun 21, 2024
54201b9
Using AssertJ recursive for test expert rules
thangqp Jul 1, 2024
e8236a8
undo set id from dto to entity
thangqp Jul 1, 2024
2f3c535
merge from main
thangqp Jul 1, 2024
0e8609f
Correct as suggest of Etienne
thangqp Jul 1, 2024
1f22f0b
Tiny correction for description
thangqp Jul 1, 2024
d11226a
Sonar Call transactional methods via an injected dependency instead o…
thangqp Jul 1, 2024
cb40b78
Enrich test for update in batch with none existing filter id
thangqp Jul 1, 2024
6c2ba17
Correct following some comments of Mathieu
thangqp Jul 3, 2024
bd50593
Merge branch 'main' into dynamic_simulation_add_some_batch_crud_endpo…
thangqp Jul 5, 2024
0713dfc
change url path in the filte-server endpoint for duplication in batch
thangqp Jul 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 39 additions & 1 deletion src/main/java/org/gridsuite/filter/server/FilterController.java
Original file line number Diff line number Diff line change
Expand Up @@ -74,9 +74,19 @@ public ResponseEntity<AbstractFilter> createFilter(@RequestParam("id") UUID filt
.body(service.createFilter(filter));
}

@PostMapping(value = "/filters/batch", consumes = MediaType.APPLICATION_JSON_VALUE)
@Operation(summary = "Create filters with provided uuids")
Mathieu-Deharbe marked this conversation as resolved.
Show resolved Hide resolved
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "Filters has been successfully created")})
Mathieu-Deharbe marked this conversation as resolved.
Show resolved Hide resolved
public ResponseEntity<List<AbstractFilter>> createFilters(@RequestBody Map<UUID, AbstractFilter> filtersToCreateMap) {
filtersToCreateMap.forEach((uuid, expertFilter) -> expertFilter.setId(uuid));
return ResponseEntity.ok()
.contentType(MediaType.APPLICATION_JSON)
.body(service.createFilters(filtersToCreateMap.values().stream().toList()));
}

@PostMapping(value = "/filters", params = "duplicateFrom")
@Operation(summary = "Duplicate a filter")
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "The filter has been successfully created"),
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "The uuid of the new filter has been successfully duplicated"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "The uuid of the new filter has been successfully duplicated"),
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "The filter with uuid filterId has been successfully duplicated"),

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, I have a slight change from your suggestion to "The filter with given id has been successfully duplicated"

@ApiResponse(responseCode = "404", description = "Source filter not found")})
public ResponseEntity<UUID> duplicateFilter(@RequestParam("duplicateFrom") UUID filterId) {
return service.duplicateFilter(filterId).map(newFilterId -> ResponseEntity.ok()
Expand All @@ -85,6 +95,17 @@ public ResponseEntity<UUID> duplicateFilter(@RequestParam("duplicateFrom") UUID
.orElse(ResponseEntity.notFound().build());
}

@PostMapping(value = "/filters/batch/duplicate")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding duplicate looks like a good way to differentiate the two /filters/batch but I don't get the fact that this has not be done for the 2 @PostMapping(value = "/filters") (the creation and the duplication have the same path).

Our rest API best practices are not very precise about extra data like this. But they state that the path should only be used to identify a data.

@etiennehomer shouldn't we modify the filters duplication signature in an other PR ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add "duplicate" in the path of duplicate endpoint, i.e. POST /filters/duplicate but we have to changer also in gridexplorer-app. I think we can change in another TS

In fact we should have :
POST /filters
POST /filters/batch
POST /filters/duplicate
POST /filters/duplicate/batch

At least I can change from /filters/batch/duplicate to /filters/duplicate/batch, what do you think??? @etiennehomer @Mathieu-Deharbe

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I think the API you propose is good Thang. So let's change POST /filters/batch/duplicate to POST /filters/duplicate/batch
And then, in another PR, let's change for unitary duplication endpoint

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DONE in 0713dfc

@Operation(summary = "Duplicate filters from provided uuids")
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "The map of sourceUuid and newUuid of duplicated filters"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "The map of sourceUuid and newUuid of duplicated filters"),
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "Filters have been successfully duplicated."),

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, I have a slight change from your suggestion to "Filters of given ids have been successfully duplicated"

@ApiResponse(responseCode = "404", description = "Source filter not found")})
public ResponseEntity<Map<UUID, UUID>> duplicateFilters(@RequestBody List<UUID> filterUuids) {
Map<UUID, UUID> uuidsMap = service.duplicateFilters(filterUuids);
return ResponseEntity.ok()
.contentType(MediaType.APPLICATION_JSON)
.body(uuidsMap);
}

@PutMapping(value = "/filters/{id}", consumes = MediaType.APPLICATION_JSON_VALUE)
@Operation(summary = "Modify a filter")
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "The filter has been successfully modified")})
Expand All @@ -97,6 +118,15 @@ public ResponseEntity<Void> changeFilter(@PathVariable UUID id, @RequestBody Abs
}
}

@PutMapping(value = "/filters/batch", consumes = MediaType.APPLICATION_JSON_VALUE)
@Operation(summary = "Modify filters in batch")
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "Filters has been successfully modified")})
Mathieu-Deharbe marked this conversation as resolved.
Show resolved Hide resolved
public ResponseEntity<List<AbstractFilter>> changeFilters(@RequestBody Map<UUID, AbstractFilter> filtersToModifyMap) {
Mathieu-Deharbe marked this conversation as resolved.
Show resolved Hide resolved
return ResponseEntity.ok()
.contentType(MediaType.APPLICATION_JSON)
.body(service.changeFilters(filtersToModifyMap));
}

@DeleteMapping(value = "/filters/{id}")
@Operation(summary = "delete the filter")
@ApiResponse(responseCode = "200", description = "The filter has been deleted")
Expand All @@ -105,6 +135,14 @@ public ResponseEntity<Void> deleteFilter(@PathVariable("id") UUID id) {
return ResponseEntity.ok().build();
}

@DeleteMapping(value = "/filters")
Mathieu-Deharbe marked this conversation as resolved.
Show resolved Hide resolved
@Operation(summary = "delete the filters")
@ApiResponse(responseCode = "200", description = "The filters have been deleted")
public ResponseEntity<Void> deleteFilters(@RequestBody List<UUID> ids) {
service.deleteFilters(ids);
return ResponseEntity.ok().build();
}

@GetMapping(value = "/filters/metadata")
@Operation(summary = "get filters metadata")
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "filters metadata"),
Expand Down
135 changes: 103 additions & 32 deletions src/main/java/org/gridsuite/filter/server/FilterService.java
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import com.powsybl.iidm.network.Network;
import com.powsybl.network.store.client.NetworkStoreService;
import com.powsybl.network.store.client.PreloadingStrategy;
import org.apache.commons.collections4.CollectionUtils;
import org.gridsuite.filter.AbstractFilter;
import org.gridsuite.filter.FilterLoader;
import org.gridsuite.filter.IFilterAttributes;
Expand All @@ -19,39 +20,11 @@
import org.gridsuite.filter.server.dto.IdsByGroup;
import org.gridsuite.filter.server.entities.AbstractFilterEntity;
import org.gridsuite.filter.server.repositories.FilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.BatteryFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.BusBarSectionFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.DanglingLineFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.GeneratorFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.HvdcLineFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.LccConverterStationFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.LineFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.LoadFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.ShuntCompensatorFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.StaticVarCompensatorFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.SubstationFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.ThreeWindingsTransformerFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.TwoWindingsTransformerFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.VoltageLevelFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.VscConverterStationFilterRepository;
import org.gridsuite.filter.server.repositories.criteriafilter.*;
import org.gridsuite.filter.server.repositories.expertfilter.ExpertFilterRepository;
import org.gridsuite.filter.server.repositories.identifierlistfilter.IdentifierListFilterRepository;
import org.gridsuite.filter.server.repositories.proxies.AbstractFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.BatteryFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.BusBarSectionFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.DanglingLineFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.GeneratorFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.HvdcLineFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.LccConverterStationFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.LineFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.LoadFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.ShuntCompensatorFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.StaticVarCompensatorFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.SubstationFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.ThreeWindingsTransformerFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.TwoWindingsTransformerFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.VoltageLevelFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.VscConverterStationFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.criteriafilter.*;
import org.gridsuite.filter.server.repositories.proxies.expertfiler.ExpertFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.identifierlistfilter.IdentifierListFilterRepositoryProxy;
import org.gridsuite.filter.server.repositories.proxies.scriptfilter.ScriptFilterRepositoryProxy;
Expand Down Expand Up @@ -81,6 +54,7 @@ public class FilterService {

private static final String FILTER_LIST = "Filter list ";
private static final String NOT_FOUND = " not found";
public static final String FILTER_UUIDS_NOT_FOUND = "Some filter uuids not found";
Mathieu-Deharbe marked this conversation as resolved.
Show resolved Hide resolved

private final Map<String, AbstractFilterRepositoryProxy<?, ?>> filterRepositories = new HashMap<>();

Expand Down Expand Up @@ -156,14 +130,28 @@ public List<AbstractFilter> getFilters(List<UUID> ids) {
.stream()
.flatMap(repository -> repository.getFilters(ids)
.stream())
.collect(Collectors.toList());
.toList();
}

@Transactional(propagation = Propagation.REQUIRED)
public <F extends AbstractFilter> AbstractFilter createFilter(F filter) {
return getRepository(filter).insert(filter);
}

@Transactional(propagation = Propagation.REQUIRED)
public List<AbstractFilter> createFilters(List<AbstractFilter> filters) {
if (CollectionUtils.isEmpty(filters)) {
return Collections.emptyList();
}

Map<AbstractFilterRepositoryProxy<?, ?>, List<AbstractFilter>> repositoryFiltersMap = filters.stream()
.collect(Collectors.groupingBy(this::getRepository));

List<AbstractFilter> createdFilters = new ArrayList<>();
repositoryFiltersMap.forEach((repository, subFilters) -> createdFilters.addAll(repository.insertAll(subFilters)));
return createdFilters;
}

@Transactional
public Optional<UUID> duplicateFilter(UUID sourceFilterId) {
Optional<AbstractFilter> sourceFilterOptional = getFilter(sourceFilterId);
Expand All @@ -177,7 +165,33 @@ public Optional<UUID> duplicateFilter(UUID sourceFilterId) {
return Optional.empty();
}

private AbstractFilterRepositoryProxy<? extends AbstractFilterEntity, ? extends FilterRepository<? extends AbstractFilterEntity>> getRepository(AbstractFilter filter) {
@Transactional
public Map<UUID, UUID> duplicateFilters(List<UUID> filterUuids) {
Mathieu-Deharbe marked this conversation as resolved.
Show resolved Hide resolved
Map<UUID, UUID> uuidsMap = new HashMap<>();

List<AbstractFilter> sourceFilters = getFilters(filterUuids);

// check whether found all
if (sourceFilters.isEmpty() || sourceFilters.size() != filterUuids.size()) {
throw new ResponseStatusException(HttpStatus.NOT_FOUND, FILTER_UUIDS_NOT_FOUND);
}

sourceFilters.forEach(sourceFilter -> {
UUID newFilterId = UUID.randomUUID();
uuidsMap.put(sourceFilter.getId(), newFilterId);
sourceFilter.setId(newFilterId);
});

Map<AbstractFilterRepositoryProxy<?, ?>, List<AbstractFilter>> repositoryFiltersMap = sourceFilters.stream()
.collect(Collectors.groupingBy(this::getRepository));

repositoryFiltersMap.forEach(AbstractFilterRepositoryProxy::insertAll);

return uuidsMap;
}

private AbstractFilterRepositoryProxy<? extends AbstractFilterEntity,
? extends FilterRepository<? extends AbstractFilterEntity>> getRepository(AbstractFilter filter) {
if (!filter.getType().equals(FilterType.CRITERIA)) {
return filterRepositories.get(filter.getType().name());
}
Expand Down Expand Up @@ -205,13 +219,70 @@ public <F extends AbstractFilter> void changeFilter(UUID id, F newFilter, String
notificationService.emitElementUpdated(id, userId);
}

@Transactional
public List<AbstractFilter> changeFilters(Map<UUID, AbstractFilter> filtersToModifyMap) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public List<AbstractFilter> changeFilters(Map<UUID, AbstractFilter> filtersToModifyMap) {
public List<AbstractFilter> updateFilters(Map<UUID, AbstractFilter> filtersToModifyMap) {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is quite complicated. As we can assume only few filters will be updated at one time, you can iterate on filtersToModifyMap and call changeFilter.
And rename changeFilter to updateFilter

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0e8609f

List<UUID> filterUuids = filtersToModifyMap.keySet().stream().toList();
List<AbstractFilter> oldFilters = getFilters(filterUuids);

// check whether found all
if (oldFilters.isEmpty() || oldFilters.size() != filterUuids.size()) {
throw new ResponseStatusException(HttpStatus.NOT_FOUND, FILTER_UUIDS_NOT_FOUND);
}

// collect filters into two lists which have repository changed or not changed
Map<Boolean, List<AbstractFilter>> filtersReposNotChangedOrChanged = oldFilters.stream().collect(Collectors
.partitioningBy(oldFilter -> getRepository(oldFilter) == getRepository(filtersToModifyMap.get(oldFilter.getId()))));
List<AbstractFilter> filtersReposNotChanged = filtersReposNotChangedOrChanged.get(Boolean.TRUE);
List<AbstractFilter> filtersReposChanged = filtersReposNotChangedOrChanged.get(Boolean.FALSE);

List<AbstractFilter> changedFilters = new ArrayList<>();

// --- perform change filters which have repository not changed --- //
Map<AbstractFilterRepositoryProxy<?, ?>, List<AbstractFilter>> repositoryFiltersMap = filtersReposNotChanged.stream()
.map(filter -> {
AbstractFilter newFilter = filtersToModifyMap.get(filter.getId());
newFilter.setId(filter.getId());
return newFilter;
})
.collect(Collectors.groupingBy(this::getRepository));
repositoryFiltersMap.forEach((repository, subFilters) ->
changedFilters.addAll(repository.modifyAll(subFilters.stream().collect(Collectors.toMap(AbstractFilter::getId, filter -> filter)))));

// --- perform change filters which have repository changed --- //
List<AbstractFilter> filtersReposChangedToModify = filtersReposChanged.stream()
.filter(filter -> filter.getType() != FilterType.SCRIPT &&
filtersToModifyMap.get(filter.getId()).getType() != FilterType.SCRIPT)
.toList();

repositoryFiltersMap = filtersReposChangedToModify.stream()
.map(filter -> {
AbstractFilter newFilter = filtersToModifyMap.get(filter.getId());
newFilter.setId(filter.getId());
return newFilter;
})
.collect(Collectors.groupingBy(this::getRepository));

// delete old filters which have repository changed
repositoryFiltersMap.forEach((repository, subFilters) ->
repository.deleteAllByIds(subFilters.stream().map(AbstractFilter::getId).toList()));
// create new filters with existing ids
repositoryFiltersMap.forEach((repository, subFilters) -> changedFilters.addAll(repository.insertAll(subFilters)));

return changedFilters;
}

public void deleteFilter(UUID id) {
Objects.requireNonNull(id);
if (filterRepositories.values().stream().noneMatch(repository -> repository.deleteById(id))) {
throw new ResponseStatusException(HttpStatus.NOT_FOUND, FILTER_LIST + id + NOT_FOUND);
}
}

public void deleteFilters(List<UUID> ids) {
Objects.requireNonNull(ids);
filterRepositories.values().forEach(repository -> repository.deleteAllByIds(ids));
Mathieu-Deharbe marked this conversation as resolved.
Show resolved Hide resolved
}

public void deleteAll() {
filterRepositories.values().forEach(AbstractFilterRepositoryProxy::deleteAll);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,7 @@ public interface FilterRepository<T extends AbstractFilterEntity> extends JpaRep

@Transactional
Integer removeById(UUID id);

@Transactional
void deleteAllByIdIn(List<UUID> ids);
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,10 @@
package org.gridsuite.filter.server.repositories.proxies;

import com.powsybl.commons.PowsyblException;

import org.gridsuite.filter.AbstractFilter;
import org.gridsuite.filter.criteriafilter.*;
import org.gridsuite.filter.server.dto.FilterAttributes;
import org.gridsuite.filter.server.entities.*;
import org.gridsuite.filter.server.entities.AbstractFilterEntity;
import org.gridsuite.filter.server.entities.criteriafilter.*;
import org.gridsuite.filter.server.repositories.FilterMetadata;
import org.gridsuite.filter.server.repositories.FilterRepository;
Expand Down Expand Up @@ -123,15 +122,30 @@ public AbstractFilter insert(AbstractFilter f) {
return toDto(getRepository().save(fromDto(f)));
}

public List<AbstractFilter> insertAll(List<AbstractFilter> filters) {
List<F> savedFilterEntities = getRepository().saveAll(filters.stream().map(this::fromDto).toList());
return savedFilterEntities.stream().map(this::toDto).toList();
}

public void modify(UUID id, AbstractFilter f) {
f.setId(id);
toDto(getRepository().save(fromDto(f)));
}

public List<AbstractFilter> modifyAll(Map<UUID, AbstractFilter> filtersToModifyMap) {
filtersToModifyMap.forEach((uuid, expertFilter) -> expertFilter.setId(uuid));
List<F> savedFilterEntities = getRepository().saveAll(filtersToModifyMap.values().stream().map(this::fromDto).toList());
return savedFilterEntities.stream().map(this::toDto).toList();
}

public boolean deleteById(UUID id) {
return getRepository().removeById(id) != 0;
}

public void deleteAllByIds(List<UUID> ids) {
getRepository().deleteAllByIdIn(ids);
}

public void deleteAll() {
getRepository().deleteAll();
}
Expand Down
Loading
Loading