biarity / sieve Goto Github PK
View Code? Open in Web Editor NEW⚗️ Clean & extensible Sorting, Filtering, and Pagination for ASP.NET Core
License: Other
⚗️ Clean & extensible Sorting, Filtering, and Pagination for ASP.NET Core
License: Other
Hello,
How you can work with a array type filters.
&filters=LikeCount>10,Author==[Brad Pitt,Mel Gibson]
I am asking for a hint.
I pass a SieveModel
with all properties set to null
to the SieveProcessor
.
In the ApplyPagination<T>
method, Skip
is applied although the pageSize value is 0.
Sieve/Sieve/Services/SieveProcessor.cs
Line 340 in 9f6163d
This is not necessary afaik.
Moving this statement within the subsequent if-test if (pageSize > 0)
would solve this issue:
Because Sorting is skipped entirely (since sieveModel.Sorts is null
), enumeration of the results throws a System.NotSupportedException
:
The method 'Skip' is only supported for sorted input in LINQ to Entities. The method 'OrderBy' must be called before the method 'Skip'.
StackTrace:
at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.OrderByLifter.PassthroughOrderByLifter.Skip(DbExpression k)
at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.OrderByLifter.Skip(DbExpressionBinding input, DbExpression skipCount)
at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.Skip(DbExpressionBinding input, DbExpression skipCount)
at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MethodCallTranslator.SkipTranslator.TranslatePagingOperator(ExpressionConverter parent, DbExpression operand, DbExpression count)
at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MethodCallTranslator.PagingTranslator.TranslateUnary(ExpressionConverter parent, DbExpression operand, MethodCallExpression call)
at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MethodCallTranslator.UnarySequenceMethodTranslator.Translate(ExpressionConverter parent, MethodCallExpression call)
at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MethodCallTranslator.SequenceMethodTranslator.Translate(ExpressionConverter parent, MethodCallExpression call, SequenceMethod sequenceMethod)
at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MethodCallTranslator.TypedTranslate(ExpressionConverter parent, MethodCallExpression linq)
at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TypedTranslator`1.Translate(ExpressionConverter parent, Expression linq)
at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TranslateExpression(Expression linq)
at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.Convert()
at System.Data.Entity.Core.Objects.ELinq.ELinqQueryState.GetExecutionPlan(Nullable`1 forMergeOption)
at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass7.<GetResults>b__6()
at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)
at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass7.<GetResults>b__5()
at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func`1 operation)
at System.Data.Entity.Core.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)
at System.Data.Entity.Core.Objects.ObjectQuery`1.<System.Collections.Generic.IEnumerable<T>.GetEnumerator>b__0()
at System.Data.Entity.Internal.LazyEnumerator`1.MoveNext()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at XXXX
Our code:
var dbQuery = ctx.Identifications.AsNoTracking();
var filtered = sieveProcessor.Apply(sieveModel, dbQuery);
return filtered.ToList();
An alternative would be to somehow specify a default Sorting property.
That way, even an empty SieveModel object would result in a sorted collection, and that can be paginated/skipped without an issue.
I worked around this by adding a manual default
if (string.IsNullOrEmpty(sieveModel.Sorts))
{
sieveModel.Sorts = "Identifier";
}
Making this part of e.g. the Fluent API would be awesome:
mapper.Property<Identification>(e => e.Identifier)
.CanFilter()
.CanSort()
.IsDefaultSortProperty(); // <------- ?
Hi,
thanks for great library. Do you have plans on extending library with 2 addidtional common functionalities that robust REST API should implement:
Post.cs
class with related property Comments and by defaultPost.cs
public int Id { get; set; }
[Sieve(CanFilter = true, CanSort = true)]
public string Title { get; set; }
[Sieve(CanFilter = true, CanSort = true)]
public int LikeCount { get; set; }
[Sieve(CanFilter = true, CanSort = true)]
public int CommentCount { get; set; }
[Sieve(CanFilter = true, CanSort = true, Name = "created")]
public DateTimeOffset DateCreated { get; set; } = DateTimeOffset.UtcNow;
// related data
public ICollection<Comment> Comments { get; set; } = new List<Comment>();
Comment.cs
public int Id { get; set; }
public string Title { get; set; }
public string Message { get; set; }
public string CommenterName { get; set; }
public DateTimeOffset DateCreated { get; set; } = DateTimeOffset.UtcNow;
and we want to include related data in results with request query string like:
GET /GetPosts
?sorts= LikeCount,CommentCount,-created // sort by likes, then comments, then descendingly by date created
&filters= LikeCount>10, Title@=awesome title, // filter to posts with more than 10 likes, and a title that contains the phrase "awesome title"
&include= Comments // include related data
&page= 1 // get the first page...
&pageSize= 10 // ...which contains 10 posts
GET /GetPosts
?sorts= LikeCount,CommentCount,-created // sort by likes, then comments, then descendingly by date created
&filters= LikeCount>10, Title@=awesome title, // filter to posts with more than 10 likes, and a title that contains the phrase "awesome title"
&include= Comments // include related data
&fields= Id, Title, LikeCount // posts with only wanted fields
&page= 1 // get the first page...
&pageSize= 10 // ...which contains 10 posts
This would make for a convenient way of isolating custom filters/sorts for different models into different classes for modularity in larger projects. Discussed in #30.
Just a suggestion, but I imagine for large sets you'd see a noticeable performance improvement by filtering the results before sorting them. The paging has to stay at the end, of course.
Please add sample projects with examples of end-to-end usage in ASP.NET Core 2.1 with:
Hi,
I enjoy the package a lot. But I have a problem, I hope to find answer to solve it. Please help me!
When I query with datetime data format, I'll get exception with sql exception conversion failed when converting date and/or time from character string
.
This is the SieveModel(with Newtonsoft.Json.JsonConvert
to get object string):
{"Filters":"datetimeColumn>2018-10-10","Sorts":null,"Page":null,"PageSize":null}
I use SQL Server Prefilter to check the sql, and I found that will be transfer to:
exec sp_executesql N'SELECT [p].[ID], [p].[datetimeColumn]
FROM XXX_table AS [p]
WHERE ([p].[datetimeColumn] > ''2018-10-10T00:00:00'')
ORDER BY (SELECT 1)
OFFSET @__p_0 ROWS',N'@__p_0 int',@__p_0=0
In above sql, I found the where
statement will add twice 「'」char. Dose there anyone have the same problem?
Thanks!
PS. My develop environment
Microsoft.AspNetCore.App 2.1.5
Sieve 2.1.5
Tried this filters=PaymentTime>=2018-12-18, PaymentTime<=2018-12-19
and it doesn't work, it's compared only 1 time.
Hi,
I'm using net core 2.2. I have created a method but when I access the end point I keep getting the same error, it works with methods that doesn't use Sieve. I commented part of the code but still the same:
(Controller)
public IActionResult GetProperties(SieveModel sieveModel)
{
string g = "a";
/*
try
{
var result = _repository.Property.GetFilteredProperties();
result = _sieveProcessor.Apply(sieveModel, result);
if(result == null)
{
return NotFound();
}
else
{
return Ok(result.ToList());
}
}
catch (Exception ex)
{*/
return StatusCode(500, "Internal server error");
//}
}
This is the implementation method GetFilteredProperties
public IQueryable<Property> GetFilteredProperties()
{
return RepositoryContext.Properties.AsNoTracking();
}
The exact error is:
{"type":"https://tools.ietf.org/html/rfc7231#section-6.5.13","title":"Unsupported Media Type","status":415,"traceId":"0HLKG5K5FVU29:00000001"}
Any ideas?
I tried implementing sorting and pagination on child resources and couldn't successfully execute it. I was wondering if I missed something or child resources are not accounted for currently
Please, implement IEquatable interface for FilterTerm.
For example:
const string FILTER_STRING = "CustomFilter==a";
var deletable = new FilterTerm { Filter = FILTER_STRING };
var filters= new List<FilterTerm>
{
new FilterTerm { Filter = FILTER_STRING }
};
Console.WriteLine(
filters.Remove(deletable)
? "DELETED!"
: "NOT DELETED =("
);
will return NOT DELETED =(
because deletable.Names
!= filters[0].Names
So, maybe should do something like this:
public class FilterTerm : IEquatable<FilterTerm>
{
public bool Equals(FilterTerm other)
{
return other.Names?.Intersect(this.Names).Count() == this.Names.Length
&& other.Operator == this.Operator
&& other.Value == this.Value;
}
}
"Sieve": {
"CaseSensitive": false,
"MaxPageSize": 15
},
MaxPageSize is only taken into consideration when pageSize is higher than expected value i.e. > 15
When page size is 0 all elements are returned bypassing pagination. By default page size is 0, setting DefaultPageSize helps only if I don't explicitly specify page size. If I explicitly set 0 in the query, all elements are returned and MaxPageSize is ignored.
Firstly, great framework! I'm really loving it so far.
When a date filter containing a timezone component (e.g. DateCreated>21T12:25:31.940+10:00
) is applied to an entity field which is backed by a datetime datatype in SQL Server, the resulting SQL query generated by EF Core uses an invalid string for the date value, like so:
SELECT [e].[Id], [e].[CommentCount], [e].[DateCreated], [e].[DateLastViewed], [e].[LikeCount], [e].[Title]
FROM [Posts] AS [e]
WHERE [e].[DateLastViewed] > '2018-05-21T12:25:31.940+10:00'
ORDER BY (SELECT 1)
OFFSET @__p_0 ROWS FETCH NEXT @__p_1 ROWS ONLY"
Because the date string can't be parsed by SQL Server (as the SQL datetime datatype does not support timezones, unlike datetimeoffset, the database engine throws a type conversion error.
Strictly speaking, this is more of a bug in EF Core rather than Sieve, but it would be relatively easy to do a workaround for this by ensuring that the LINQ expressions that SieveProcessor generates match with those generated from the more mainstream Queryable.Where
method (i.e. Posts.Where(p => p.DateLastViewed > date)
)
For comparison, here's the debug view from a linq expression generated by sieve:
.Call System.Linq.Queryable.Take(
.Call System.Linq.Queryable.Skip(
.Call System.Linq.Queryable.Where(
.Call Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.AsNoTracking(.Constant<Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1[SieveTests.Entities.Post]>(Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1[SieveTests.Entities.Post]))
,
'(.Lambda #Lambda1<System.Func`2[SieveTests.Entities.Post,System.Boolean]>)),
0),
10)
.Lambda #Lambda1<System.Func`2[SieveTests.Entities.Post,System.Boolean]>(SieveTests.Entities.Post $e) {
$e.DateLastViewed != .Constant<System.DateTime>(22/05/2018 12:25:31 PM)
}
And this is the same expression, generated using
Posts.Where(p => p.DateLastViewed > date)
:
.Call System.Linq.Queryable.Where(
.Call System.Linq.Queryable.Take(
.Call System.Linq.Queryable.Skip(
.Call Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.AsNoTracking(.Constant<Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1[SieveTests.Entities.Post]>(Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1[SieveTests.Entities.Post]))
,
0),
10),
'(.Lambda #Lambda1<System.Func`2[SieveTests.Entities.Post,System.Boolean]>))
.Lambda #Lambda1<System.Func`2[SieveTests.Entities.Post,System.Boolean]>(SieveTests.Entities.Post $p) {
$p.DateLastViewed > .Constant<SieveTests.Controllers.PostsController+<>c__DisplayClass3_0>(SieveTests.Controllers.PostsController+<>c__DisplayClass3_0).date
}
This in turn generates the following SQL query, which passes the correct date value as a parameter:
SELECT [t].[Id], [t].[CommentCount], [t].[DateCreated], [t].[DateLastViewed], [t].[LikeCount], [t].[Title]
FROM (
SELECT [p].[Id], [p].[CommentCount], [p].[DateCreated], [p].[DateLastViewed], [p].[LikeCount], [p].[Title]
FROM [Posts] AS [p]
ORDER BY (SELECT 1)
OFFSET @__p_0 ROWS FETCH NEXT @__p_1 ROWS ONLY
) AS [t]
WHERE [t].[DateLastViewed] > @__date_2"
I will be submitting a pull request shortly with a fix for this issue.
Hi guys,
Was wondering if its possible to achieve this without dependency to EF? Would it be possible to just derive the SQL from the filters and annotated models? Then I can use plain SqlConnection and DataReaders for getting the object from any SQL92 source.
Appreciate your feedback.
Br, Rodel
Is there any way to workaround the comma limitation?
Don't forget to remove commas (,), brackets ((, )), and pipes (|) from any {Value} fields
I found that Sieve will flow an error if I try to use CustomFilter and don't add Sieve attributes and no CustomSieveProcessor
e.g. I got this custom filter
public class SieveCustomFilterMethods : ISieveCustomFilterMethods
{
public IQueryable<Domain.User.User> Roles(IQueryable<Domain.User.User> source, string op, string[] values) // The method is given the {Operator} & {Value}
{
var result = source;
foreach (var v in values)
result = source.Where(x => x.UserRoles.Any(r => r.Role == v));
return result; // Must return modified IQueryable<TEntity>
}
}
And this UserClass
public class User
{
public Guid UserId { get; set; }
public string Email { get; set; }
public IEnumerable<UserRole> UserRoles { get; set; }
public IEnumerable<string> Roles => UserRoles.Select(r => r.Role);
}
If I don't add [Sieve(CanFilter=true)]
to the User.UserRoles
, and call the API with ?filters=roles==admin
The return result will be filter with UserRoles that contain "Admin"
BUT! processor.Apply
will keep throwing the following exception without breaking the filter
System.Collections.Generic.KeyNotFoundException: 'The given key 'Domain.User.User' was not present in the dictionary.'
Then I tried adding a custom SieveProcessor while keeping my custom filter
public class ApplicationSieveProcessor : SieveProcessor
{
public ApplicationSieveProcessor(
IOptions<SieveOptions> options)
: base(options)
{
}
protected override SievePropertyMapper MapProperties(SievePropertyMapper mapper)
{
mapper.Property<Domain.User.User>(x => x.UserRoles)
.CanFilter();
return mapper;
}
}
Then, calling ?filters=roles==admin
will cause another exception that break the filter
Sieve.Exceptions.SieveMethodNotFoundException: 'roles not found.'
If I change the MapProperties
to
protected override SievePropertyMapper MapProperties(SievePropertyMapper mapper)
{
mapper.Property<Domain.User.User>(x => x.Roles)
.CanFilter();
return mapper;
}
This will not trigger any filter at all.
Is there a way to keep CustomFilter while adding FluentAPI properly?
Is there in plans to write frontend typed client for your library?
I already wrote something like this but without "OR" logic, because I know you want to change the parser and it seems incorrect now. Maybe it will be
filter=((my_param_1==1)&&(my_param_2==2))||(my_param_3==3)
, 'cause I can't write right filter for situation where I need object's list with several values for one property:
filter=id==(1,2,3,4)
, or I didn't find this one. I know, I can write my own parser, but I want it out of the box because it's a base ability :)
Hi,
Have you considered adding some kind of paging info on Apply
result? It's common pattern that we need Page, PageSize and TotalCount of filtered items. The only option now is to call Apply
2 times (one with turned off paging) to get TotalCount, but that's highly unefficient since there is no caching of expressions and reflections in library (that might be another case of issue though).
I think the api could look like:
PagingInfo pagingInfo = new PagingInfo();
var query = query.Apply(sieveModel, query, pagingInfo);
I don't really like the out object passed to function. But the alternative is maybe to return object from Apply
? But that's pretty big breaking change. Please let me know what you think.
EDIT: I'm more than happy to create PR with this as soon as we can figure out how API could look like :)
I am using this great library. One thing that causes problems is that UTC formatted times in filter are parsed as Local DateTimes. The following sample reproduced this behavior:
class Program
{
static void Main(string[] args)
{
SieveProcessor processor = new SieveProcessor(new SieveOptionsAccessor());
var createdAtUtc = DateTime.UtcNow;
IQueryable<Entity> entities = new[]{
new Entity { CratedAtUtc = createdAtUtc }
}.AsQueryable();
var model = new SieveModel()
{
Filters = $"CratedAtUtc=={createdAtUtc.ToString("o")}"
};
var result = processor.Apply(model, entities).ToList();
//should print 1, but prints 0
Console.WriteLine($"Count: {result.Count}");
}
}
class Entity
{
[Sieve(CanFilter = true, CanSort = true)]
public DateTime CratedAtUtc { get; set; }
}
public class SieveOptionsAccessor : IOptions<SieveOptions>
{
public SieveOptions Value { get; }
public SieveOptionsAccessor()
{
Value = new SieveOptions()
{
ThrowExceptions = true
};
}
}
With Local DateTimeKind it works properly.
I try to use the default code using ASP.NET Core 2.1 Web API and anytime I put SieveModel as a post parameter I get "The Input was not valid." as a result.
[HttpGet]
public ActionResult<IEnumerable<Publisher>> Get(SieveModel sieveModel)
{
var data = this._repository.List();
return _sieveProcessor.Apply(sieveModel, data).ToArray();
}
I think it would be reasonable to use IOptionsSnapshot
instead of IOptions
to be able to change configuration without the need to restart the server.
When you have classes like the example below and want to filter/sort on the column names "updated_at" and "created_at" it doesn't work.
I use this GET request as an example:
Works:
GET base_url/api/v1/assets?sorts=UpdatedAt
Does not work:
GET base_url/api/v1/assets?sorts=updated_at
updated_at is the property that is exposed in the json body, so it makes more sense for an end-user to utilise this name.
public class Asset
{
[JsonIgnore]
public int Id { get; set; }
[Sieve(CanFilter = true, CanSort = true)]
[JsonProperty(PropertyName = "created_at")]
public DateTime CreatedAt { get; set; }
[Sieve(CanFilter = true, CanSort = true)]
[JsonProperty(PropertyName = "updated_at")]
public DateTime UpdatedAt { get; set; }
}
I followed your readme and having this issue. What might cause that ?
var result = _context.TblObjectOutTemplateItem.AsNoTracking();
result = _sieveProcessor.ApplyAll(item, result);
and my model is simple as your test models
Actual behaviour: When filtering a DateTime field with "createdat > 2018", a FormatException is thrown ("FormatException: String was not recognized as a valid DateTime.")
Expected behaviour: Sieve should catch the FormatException and throw a sieve exception.
This may also happen with integers or other data types, although I haven't tested or checked.
Using Sieve 1.3.92.
It seems the filter operators are not parsed in the correct order:
Filter Begin>2018-04-02
works but Begin>=2018-04-02
throws a parse exception (I assume integers will have the same issue, not only DateTime objects).
System.FormatException: =2018-04-02 is not a valid value for DateTime. ---> System.FormatException: String was not recognized as a valid DateTime.
at System.DateTimeParse.Parse(String s, DateTimeFormatInfo dtfi, DateTimeStyles styles)
at System.ComponentModel.DateTimeConverter.ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, Object value)
--- End of inner exception stack trace ---
at System.ComponentModel.DateTimeConverter.ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, Object value)
at Sieve.Services.SieveProcessor.ApplyFiltering[TEntity](ISieveModel`2 model, IQueryable`1 result, Object[] dataForCustomMethods)
Hello,
I started using Sieve library a few days ago but I encountered some problems when working with objects / nested values as filter subjects. For instance, consider the following configuration in the FluentApi:
public override void Configure(SievePropertyMapper mapper)
{
mapper.Property<BackOfficeUser>(u => u.Email)
.CanFilter();
mapper.Property<BackOfficeUser>(u => u.UserName)
.CanFilter();
mapper.Property<BackOfficeUser>(u => u.FirstName)
.HasName(nameof(BackOfficeUser.FirstName).PascalToCamelCase())
.CanFilter();
mapper.Property<BackOfficeUser>(u => u.LastName)
.HasName(nameof(BackOfficeUser.LastName).PascalToCamelCase())
.CanFilter();
mapper.Property<BackOfficeUser>(u => u.PhoneNumber)
.HasName(nameof(BackOfficeUser.PhoneNumber).PascalToCamelCase())
.CanFilter();
}
The names are being specified for the FullName and PhoneNumber just for consistency when querying from the client. There are no custom filter methods defined for that entity. Here is how the FirstName, LastName and PhoneNumber properties are defined:
public Name FirstName { get; private set; }
public Name LastName { get; private set; }
public new PhoneNumber PhoneNumber { get; private set; }
The Name value object:
public class Name : ValueObject
{
public static int MaxLength = 50;
private Name() {}
public string Value { get; private set; }
public static Result<Name> Create(Maybe<string> name)
{
return name
.ToResult(KernelDomainMessages.InvalidName)
.Ensure(x => !string.IsNullOrWhiteSpace(x), KernelDomainMessages.InvalidName)
.Ensure(HasValidLength, KernelDomainMessages.InvalidName)
.Map(x => new Name {Value = x});
}
public static implicit operator string(Name name)
{
return name.Value;
}
public static Name Empty => new Name { Value = "" };
private static bool HasValidLength(string value)
{
EnsureArg.IsNotEmpty(value);
return value.Trim().Length <= MaxLength;
}
protected override IEnumerable<object> GetEqualityComponents()
{
yield return Value;
}
public override string ToString()
{
return Value;
}
}
The PhoneNumber is just another ValueObject which contains phone numbers specific domain logic. The point here is that my domain entity (BackOfficeUser) is modeled using value objects, which are concrete types defined by me, yet not primitive types like string (as Sieve is expecting). So the problems that I encounter are:
2.Sieve fails when converting string value from SieveModel to my Name type. This could be solved if I define a cast operator in my Name class, but I really don't want to do that since I want to enforce this Value Object to always have a valid state, hence a cast from any string would violate this.
But as you have seen, the value objects described above have a "Value" property which is a string. So I may use it for filtering, right? So my second though was this:
public override void Configure(SievePropertyMapper mapper)
{
mapper.Property<BackOfficeUser>(u => u.Email)
.CanFilter();
mapper.Property<BackOfficeUser>(u => u.UserName)
.CanFilter();
mapper.Property<BackOfficeUser>(u => u.FirstName.Value)
.HasName(nameof(BackOfficeUser.FirstName).PascalToCamelCase())
.CanFilter();
mapper.Property<BackOfficeUser>(u => u.LastName.Value)
.HasName(nameof(BackOfficeUser.LastName).PascalToCamelCase())
.CanFilter();
mapper.Property<BackOfficeUser>(u => u.PhoneNumber.Value)
.HasName(nameof(BackOfficeUser.PhoneNumber).PascalToCamelCase())
.CanFilter();
}
But this fails when Sieve tries to evaluate the property given in the expression at filter time. It fails with an error like "Property 'Value' is not part of type {NameSpace}.BackOfficeUser". It fails because I'm using nested values instead of direct usage.
What I've considered:
I considered creating separate filter methods for these value objects, but it doesn't work since they are part of the model and not entities as a whole.
I tried to create generic custom filter methods, like :
public IQueryable<TEntity> Name<TEntity>(IQueryable<TEntity> entities, string op, string value)
where TEntity : IOwnsFirstName, IOwnsLastName
{
//// filter specific code here
return entities;
}
But this doesn't get recognized by Sieve when looking for custom filter methods for the fiven
Possible solutions:
mapper.Property<BackOfficeUser>(u => u.LastName)
.HasValue(ln => ln.Value) // Or .HasValue<Name>(ln => ln.Value)
.HasName(nameof(BackOfficeUser.LastName).PascalToCamelCase())
.CanFilter();
I think this problem also occurs when trying to sort complex objects / nested values. It would be nice if Sieve could support that.
After updating to 2.2.0, I noticed that filters that contain operations no longer work. Filters with no operations seem to work fine though. I downloaded the latest commit of Sieve and ran SieveTests. The default custom filter implementation for Post
only uses IsNew
without taking in any operations/values.
I modified the filter to take in an operation and value:
public IQueryable<Post> IsNew(IQueryable<Post> source, string op, string value)
=> source.Where(p => p.LikeCount < Int32.Parse(value) && p.CommentCount < 5);
When you send a request using http://localhost:65136/api/posts/GetAllWithSieve?filters=isnew==100
, the filter is not used. I noticed that ApplyCustomMethod
in the class SieveProcessor
uses reflection and calls below:
try
{
result = customMethod.Invoke(parent, parameters)
as IQueryable<TEntity>;
}
customMethod.Invoke throws an exception and below is the stacktrace:
{System.ArgumentException: Object of type 'System.String[]' cannot be converted to type 'System.String'.
at System.RuntimeType.TryChangeType(Object value, Binder binder, CultureInfo culture, Boolean needsSpecialCast)
at System.Reflection.MethodBase.CheckArguments(Object[] parameters, Binder binder, BindingFlags invokeAttr, CultureInfo culture, Signature sig)
at System.Reflection.RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
at Sieve.Services.SieveProcessor`3.ApplyCustomMethod[TEntity](IQueryable`1 result, String name, Object parent, Object[] parameters, Object[] optionalParameters) in C:\Users\VR\Downloads\Sieve-master\Sieve-master\Sieve\Services\SieveProcessor.cs:line 376
at Sieve.Services.SieveProcessor`3.ApplyFiltering[TEntity](TSieveModel model, IQueryable`1 result, Object[] dataForCustomMethods) in C:\Users\VR\Downloads\Sieve-master\Sieve-master\Sieve\Services\SieveProcessor.cs:line 218
at Sieve.Services.SieveProcessor`3.Apply[TEntity](TSieveModel model, IQueryable`1 source, Object[] dataForCustomMethods, Boolean applyFiltering, Boolean applySorting, Boolean applyPagination) in C:\Users\VR\Downloads\Sieve-master\Sieve-master\Sieve\Services\SieveProcessor.cs:line 127}
I reverted my package to 2.1.5 and its working but 2.2.0 does not.
Question:
Is this only work with IEnumerable?
I do not want to change the domain entities using a decorator.
I want something like this,
//dynamically load all configuration
//System.Type configType = typeof(LanguageMap); //any of your configuration classes here
var typesToRegister = Assembly.GetExecutingAssembly().GetTypes()
.Where(type => !string.IsNullOrEmpty(type.Namespace))
.Where(type => type.BaseType != null && type.BaseType.IsGenericType &&
type.BaseType.GetGenericTypeDefinition() == typeof(EntityTypeConfiguration<>));
foreach (var type in typesToRegister)
{
dynamic configurationInstance = Activator.CreateInstance(type);
modelBuilder.ApplyConfiguration(configurationInstance);
}
//...or do it manually below. For example,
//modelBuilder.Configurations.Add(new LanguageMap());
Hi,
As a lot of the GET methods will use SieveModel as the input parameter, is there any example documentation on how to effectively work with Swagger/Swashbuckle? When I added Swagger, the GetAllWithSiev action in the test project produces all of the model parameters in the swagger UI.
Could you provide some references or examples on how to provide references for how the sorts/filters/pagination parameters work with swagger?
Hi there,
I'm using UnitOfWork and wrote this method but I'm getting this error:
The offset specified in a OFFSET clause may not be negative.
public IQueryable<Note> GetAll(SieveModel sieveModel)
{
var result = UnitOfWork.Select<Note>()
.Where(s => (s is Note) && s.IsDeleted == false && s.NoteType == "Note" && s.UserUid == _currentUser.Uid)
.AsNoTracking();
result = _sieveProcessor.ApplyAll(sieveModel, result);
return result;
}
Is this a bug?
Thanks.
Is there or is it planned to get total number of records found by filter?
Thanks
Problem
Trying to use Sieve with MediatR & CQRS, so that sieveModel is passed from Controller to Query class then handled by QueryHandler
Expected Behaviour
ISieveProcessor
should be injected properly in QueryHandler
Current Behaviour
Failed to find ISieveProcessor
in QueryHandler
although it is injected in Startup.cs
as below
// Add Sieve
services.AddScoped<ISieveProcessor, ApplicationSieveProcessor>();
services.Configure<SieveOptions>(Configuration.GetSection("Sieve"));
Exception Message
Error constructing handler for request of type
MediatR.IRequestHandler 2[Northwind.Application.Products.Queries.GetAllProducts.GetAllProductsQuery,Northwind.Application.Products.Queries.GetAllProducts.ProductsListViewModel].
Register your handlers with the container. See the samples in GitHub for examples.
StackTrace
at MediatR.Internal.RequestHandlerBase.GetHandler[THandler](ServiceFactory factory)
at MediatR.Internal.RequestHandlerWrapperImpl`2.<>c__DisplayClass0_0.<Handle>g__Handler|0()
at MediatR.Pipeline.RequestPostProcessorBehavior`2.Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate`1 next)
at MediatR.Pipeline.RequestPreProcessorBehavior`2.Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate`1 next)
at Northwind.Application.Infrastructure.RequestPerformanceBehaviour`2.Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate`1 next) in C:\Users\amery\Desktop\MediatR CQRS & Sieve\NorthwindTraders-master\Northwind.Application\Infrastructure\RequestPerformanceBehaviour.cs:line 25
at MediatR.Pipeline.RequestPreProcessorBehavior`2.Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate`1 next)
at Northwind.WebUI.Controllers.ProductsController.GetAll(SieveModel sieveModel) in C:\Users\amery\Desktop\MediatR CQRS & Sieve\NorthwindTraders-master\Northwind.WebUI\Controllers\ProductsController.cs:line 21
at lambda_method(Closure , Object )
at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()
at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at System.Threading.Tasks.ValueTask`1.get_Result()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextExceptionFilterAsync()
Example
I Created This Gist that replicate the issue here:
https://gist.github.com/yehia2amer/869984e1837b4683767035964aac680d
This is based on NorthwindTraders Sample.
Full Project Example
https://github.com/yehia2amer/NorthwindTraders_MediatR_Sieve
I would like to be able to (optionally) explicitly specify which properties are used for the queries instead of using the Sieve attribute. Ideally, this "specification" would also allow the definition of a mapping between a query term to an actual property on the model class.
My arguments supporting this request:
Here is an example of how the changed call to SieveProcessor could look like:
` var result = _dbContext.Posts.AsNoTracking();
var sieveProperties = new[]
{
SieveProperty<Post>.For(_ => _.Title, Allow.Filter, "name"),
SieveProperty<Post>.For(_ => _.CommentCount, Allow.SortAndFilter),
SieveProperty<Post>.For(_ => _.LikeCount, Allow.Sort),
SieveProperty<Post>.For(_ => _.DateCreated, Allow.SortAndFilter),
};
result = _sieveProcessor.ApplyAll(sieveModel, result, sieveProperties);
return Json(result.ToList());
I have implemented a draft of the necessary changes and will open a PR. Feedback is welcome!
BTW: Great work so far!
The mentioned operators are case sensitive. I know that it is possible to implement custom operators but I would claim that most scenarios require a case insensitive comparison.
Thanks for sharing this project 👍
Why do you target .NET Core 2 and not .NET Standard 2? I'd like to use this for ASP.NET Core project that targets .NET Framework 4.7.
Feature to specify the maximum page size to prevent bad requests. For example with pageSize=9999999.
I have been using Sieve for quite some time now and it works great. I was wondering if it is possible to do or logic on same property with different values.
i.e. Book { Title, Author }
Is there a way to filter on books with title "Gulliver's Travels" OR "The Society of Mind"],
Thank you
I have a class that is setup as below:
Event
public class Event
{
public Guid Id { get; set; }
public string Message { get; set; }
public Guid UserId { get; set; }
public User CreatedBy { get; set; }
}
User Class
public class User
{
public Guid Id { get; set; }
public string Email { get; set; }
}
My outer facing model is different from my Event entity as I don't want the user to have access to all the fields. So its setup as:
public class EventViewDto
{
public Guid Id { get; set; }
public string Message { get; set; }
public string UserEmail { get; set; } //user email is mapped from User.Email when returning data to client
}
I can't get Sieve to filter on the navigation property with below setup:
protected override SievePropertyMapper MapProperties(SievePropertyMapper mapper)
{
mapper.Property<Event>(p => p.CreatedBy.Email)
.CanFilter()
.CanSort()
.HasName("UserEmail");
}
The only way to get it to work is by writing a custom filter like below:
public class EventCustomFilterMethods: ISieveCustomFilterMethods
{
public IQueryable<Event> OwnerEmail(IQueryable<Event> source, string op, string value)
=> source.Where(p => p.CreatedBy.Email.Contains(value));
}
The issue with this approach is that if you have multiple classes where such transformation is taking place, you have to write a custom filter for each property which is being transformed from Entity to outer facing user model. Is it possible to use a generic version of this custom filter so that it can be applied to multiple classes that share the same property?
(I think this is the issue @davidnmbond mentioned in #8 )
In SieveModel.cs where the filter expression is parsed only more AND-conditions are added to the list of filter items.
I think the issue happened because of a typo in the unit test here. I assume the value for Title
should be "D"
not "3"
- then the test fails as expected.
Newly added .editorconfig file has not been applied yet.
While attempting to make a custom search filter that was shared across multiple models I found that once a specific filter name was used multiple times that filter would fail.
For example, in your tests you have an IsNew filter defined on your Posts.
If you create an IsNew filter on your Comments (and add a DateCreated field on Comments to allow the filter to function) Sieve will not be able to use this filter properly.
It does not attempt to find the IsNew filter defined for a given type, it simply searches for a method matching that name.
I've submitted a fix as PR #29 but thought I should create an Issue in case you wanted to solve it differently.
#29 added code that searches for custom filter/sort method while assuming the type and number of parameters passed to these methods. In reality this will not always be certain since custom filter and custom sort methods accept different arguments. There is also the feature "dataForCustomMethods" where custom data can be passed into SieveProcessor.Apply
, adding more unpredictable parameters.
A simple way to fix this is to only check that the type of the first argument matches (then maybe fail later if the others don't without explicitly checking first). Unfortunately there were no tests for custom sorting methods which is why this went under the radar for quite some time now.
Would it be possible to interpret having no values for a filter as a single blank string?
for example...
api/posts?Filters=LikeCount>5,Title==
Something about this URL causes the filters to be ignored completely. I get all results.
I do have a workaround, though. I'm using a URL similar to the following...
api/posts?Filters=LikeCount>5,Title==|
This will return all posts with LikeCount > 5 and a blank Title. The pipe here (with no characters to either side) comes in as two values: both blank strings. The actual query is probably: all posts with LikeCount > 5 and a blank Title and a blank Title
Any use of the pipe character works when it comes to blank strings...
Title1| = "Title1" or ""
|Title1 = "" or "Title1"
Title1||Title2 = "Title1" or "" or "Title2"
I know that it is possible to implement custom operators but I would claim that most scenarios require a "not contains" comparison.
Sieve is quite limited when it comes to complex conditions. Make it possible to use Expression.Or
so something like filters=name==fu manchu||age=70,country=China
When an invalid filter expression or a filter for a non-filterable property is passed to the SieveProcessor, it is simply ignored. With this behavior, it is not possible to give helpful feedback to consumers of an API, e.g. "400 - Invalid filters specified" or even "Filtering for 'description' is not allowed".
What would you think about throwing SieveExceptions in such cases?
I have been playing with Sieve over the last few days and it is really cool.
However, one feature that I think is missing, is the ability to express filters like this:
"If property1 or property2 contain ABC"
I imagine that the {Name}
part of the filter expressions could be extended like this:
(property1,property2)@=ABC
What do you think about this?
Hi!
Great project! I've noticed that attempting to filter on fields of type int?
breaks Sieve.
On my model I have:
[Sieve(CanFilter = true, CanSort = true)]
public int? CategoryId { get; set; }
I'd expect to be able to &filters=CategoryId==5
but instead that results in all of the results being sent instead of my filtered, paginated result.
Hi!
I have an implementation of ISieveCustomFilterMethods for a custom search filter named contains
.
If I use the query string ?filters=contains@=robots,categoryId==12
the results are properly filtered with the search and the category.
If I switch the order of the standard filter and the custom filter: ?filters=categoryId==12,contains@=robots
Sieve fails to filter and I get all of the results instead.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.