tcsenpai/pensieve
1
0
Fork 0
mirror of https://github.com/tcsenpai/pensieve.git synced 2025-06-06 03:05:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
448 Commits 1 Branch 62 Tags 90 MiB
Commit Graph

249 Commits

Author SHA1 Message Date
arkohut
d98eab9ebd fix: ocr fetch bug fix 2024-09-11 23:20:42 +08:00
arkohut
167fd31053 feat: add builtin plugins for default library 2024-09-10 15:47:36 +08:00
arkohut
fca387b22d feat: support bind plugin by name 2024-09-10 13:55:52 +08:00
arkohut
5882950c39 feat(cli): add extra args 2024-09-10 12:39:08 +08:00
arkohut
7f78eb1a4a chore: update yaml generate config 2024-09-10 11:54:56 +08:00
arkohut
6bbf1a8a68 fix: use modelscope for sub config 2024-09-10 01:59:37 +08:00
arkohut
378e5bf445 feat: support modelscope 2024-09-10 01:32:00 +08:00
arkohut
241911d1d2 feat(index): use different bs for embedding 2024-09-10 00:58:57 +08:00
arkohut
f5aae87f40 feat: use concurrency of 1 by default 2024-09-10 00:54:20 +08:00
arkohut
41c7e136d9 fix: skip flash attn 
https://huggingface.co/microsoft/Florence-2-base-ft/discussions/13#66836a8d67d2ccf03a96df8d
 2024-09-10 00:31:56 +08:00
arkohut
a30fe62bc3 fix: yaml parse related 2024-09-09 23:39:04 +08:00
arkohut
3912d165f6 Revert "fix: skip flash attn https://huggingface.co/microsoft/Florence-2-base-ft/discussions/13#66836a8d67d2ccf03a96df8d" 
This reverts commit 5b1194f1bc6ad055fe69a9a2207ef1c31cc73721.
 2024-09-09 23:27:40 +08:00
arkohut
5b1194f1bc fix: skip flash attn https://huggingface.co/microsoft/Florence-2-base-ft/discussions/13#66836a8d67d2ccf03a96df8d 2024-09-09 22:26:48 +08:00
arkohut
6107c22def fix: support parse secret 2024-09-09 22:26:26 +08:00
arkohut
81ba2cd2d2 feat: add default library and command shortcuts for it 2024-09-09 20:58:06 +08:00
arkohut
7e43bc0861 feat(ml_backend): move florence 2 as a default vlm plugin 2024-09-09 20:30:58 +08:00
arkohut
d7e6c32e86 feat: init default library 2024-09-09 19:57:07 +08:00
arkohut
69aca0153a feat: make embedding a default plugin 2024-09-09 19:43:17 +08:00
arkohut
dbbc2792ef feat: enable baisc auth 2024-09-06 18:18:18 +08:00
arkohut
e062636c08 feat(ocr): update import flow 2024-09-05 18:15:27 +08:00
arkohut
80c261ba8a feat: add ml backend server 2024-09-03 23:37:23 +08:00
arkohut
a9f34b09d4 feat(index): do not add ocr for embedding 2024-09-03 18:36:08 +08:00
arkohut
e99792a974 feat(vlm): force jpeg option 2024-09-03 18:35:41 +08:00
arkohut
2b2d616775 feat(ocr): support local paddleocr 2024-09-01 22:40:20 +08:00
arkohut
d3b45ad197 feat: enable config.yaml 2024-09-01 19:01:49 +08:00
arkohut
1a08a44a4d fix(index): ignore empty text for embedding 2024-08-29 21:38:01 +08:00
arkohut
2135b11f6c chore: hidden vector content for search param log 2024-08-29 21:16:36 +08:00
arkohut
7fbb05e0cd feat: add force indexing 2024-08-29 21:16:15 +08:00
arkohut
2f844eae80 chore: typo 2024-08-29 19:00:57 +08:00
arkohut
49879ad20e fix: use global import 2024-08-29 18:02:51 +08:00
arkohut
a448fd0c9a feat: support initialization 2024-08-29 17:47:14 +08:00
arkohut
0970e73da5 feat: make server host configurable 2024-08-29 17:01:02 +08:00
arkohut
75bb7aeef9 feat: update default config for embedding and vlm 2024-08-29 16:49:44 +08:00
arkohut
9db92908a6 feat(indexing): do not repeatly indexing updated entity by last_scan_at 2024-08-29 13:57:46 +08:00
arkohut
439adfa955 feat: support cascade delete entity 2024-08-29 13:24:23 +08:00
arkohut
16ef927c38 feat(indexing): manully control embedding generation 2024-08-29 13:24:00 +08:00
arkohut
abfc2814cf feat: make server port configurable 2024-08-29 13:11:26 +08:00
arkohut
290636ca08 feat(indexing): make some fields optional 2024-08-27 23:34:57 +08:00
arkohut
b10b080800 fix(ollama): make vlm plugin support ollama 2024-08-25 16:49:48 +08:00
arkohut
67a5e10d3e feat(plugins): add ocr as build plugin 2024-08-25 16:49:48 +08:00
arkohut
ec7ba1f989 feat(plugin): allow unbind plugin 2024-08-25 16:49:48 +08:00
arkohut
cead5d9755 feat(plugin): add vlm plugin 2024-08-25 16:49:48 +08:00
arkohut
4b9b2ce8e1 fix(typesense): no sort for tags 2024-08-23 23:25:17 +08:00
arkohut
420d972513 fix(webp): failed to save metadata for webp 2024-08-23 18:25:19 +08:00
arkohut
0d371d759e fix: use str instead of path 2024-08-22 10:00:20 +08:00
arkohut
9a38b0c17d fix: update utils import 2024-08-21 23:22:58 +08:00
arkohut
15972b9c0f feat: use utiles for read metadata 2024-08-21 21:21:32 +08:00
arkohut
b3e73e4c40 feat(screen-recorder): generate video from webp 2024-08-21 21:21:32 +08:00
arkohut
b53d816b17 feat(typesense): use ollama bge m3 2024-08-21 08:48:55 +08:00
arkohut
1d4faf2652 fix: skip system files in screenshots folder 2024-08-21 08:35:20 +08:00
arkohut
9267fdc018 feat(indexing): add sort for fields 2024-08-21 00:12:54 +08:00
arkohut
11e447bcbb feat(indexing): support facet 2024-08-21 00:12:31 +08:00
arkohut
2cf75bee7f feat(index): extract collection name to settings 2024-08-20 18:35:35 +08:00
arkohut
f0dfaf387e feat(index): add date month year for index 2024-08-20 13:18:34 +08:00
arkohut
1d10b0ef7b fix(indexing): do not check entity exists when delete index 2024-08-20 13:18:05 +08:00
arkohut
ef2022fa10 feat(screen-recorder): insert tag of app name 2024-08-20 13:17:22 +08:00
arkohut
01a294a70a feat: add active_app from active_window when scanning 2024-08-20 13:12:25 +08:00
arkohut
44f015d3fe fix: handle none response exception 2024-08-16 23:10:32 +08:00
arkohut
95acfdd174 feat: stop print metadata when read it 2024-08-14 19:44:56 +08:00
arkohut
9640e00d54 feat(cli): return lib info after add folder 2024-08-14 19:43:29 +08:00
arkohut
39887c827f feat: add more indices 2024-08-14 12:20:32 +08:00
arkohut
2314ad544e fix(index): can not support number and string at the same field 2024-08-14 12:14:31 +08:00
arkohut
fc95061672 feat(metadata): support create tags and metadata when create new entity 2024-08-14 11:31:50 +08:00
arkohut
016264f13f feat: support extract raw image from video 2024-08-13 15:35:48 +08:00
arkohut
0fff358e4d feat: add http timeout 2024-08-08 20:05:14 +08:00
arkohut
5285831d03 feat: support filter by folders and libraries 2024-07-31 18:58:50 +08:00
arkohut
025879792c fix: tolerate none response exception 2024-07-31 18:57:56 +08:00
arkohut
d0cfd91f5a feat(indexing): add time filter 2024-07-31 00:41:29 +08:00
arkohut
83bd59ca80 fix: add logconfig 2024-07-30 15:55:56 +08:00
arkohut
58ab45c100 feat(scan): tolerate add entity failed 2024-07-29 19:18:28 +08:00
arkohut
18d30e6567 feat: show update last_scan_at for any update 2024-07-29 18:25:24 +08:00
arkohut
7e7fa02e23 feat: use bulk upsert when indexing 2024-07-29 17:50:29 +08:00
arkohut
2794e6b7b4 feat: show full progress when index a folder 2024-07-29 16:50:40 +08:00
arkohut
7029ebc5b8 fix: wrong datetime 2024-07-29 16:39:31 +08:00
arkohut
6d18a3aa05 fix: retry and try catch exception when update entity 2024-07-29 16:09:06 +08:00
arkohut
873034c76b feat(scan): batching check file exists 2024-07-26 18:35:39 +08:00
arkohut
badbfd70bc feat(scan): support batching check file exists 2024-07-26 18:34:59 +08:00
arkohut
4a90d86c16 feat(scan): give the entity count when scan lib to delete entities 2024-07-26 17:14:30 +08:00
arkohut
e7be69f0ef feat: return total count in header when list foler entities 2024-07-26 17:13:42 +08:00
arkohut
4e1ce5ef96 feat: support indexing for specific folders 2024-07-26 16:27:01 +08:00
arkohut
506e5eedd0 feat(scan): support concurrent webhook trigger 2024-07-22 16:34:11 +08:00
arkohut
3855daf5b4 feat(server): update logging format 2024-07-22 16:32:55 +08:00
arkohut
c0476cbfa2 feat(scan): change timeout to 60s 2024-07-18 23:28:44 +08:00
arkohut
c997244577 feat(schema): get metadata by key 2024-07-18 23:25:47 +08:00
arkohut
f06fe7fd6c feat(search): update search parameters 2024-07-18 00:18:53 +08:00
arkohut
4ac3b78a68 feat: add alembic for migration 2024-07-17 14:28:01 +08:00
arkohut
b7e4853e26 chore: show more info when startup 2024-07-17 14:16:30 +08:00
arkohut
a28fa41153 feat(tag): add a patch tag api 2024-07-15 00:03:05 +08:00
arkohut
409e5c245f feat(cmd): scan for specific folders 2024-07-14 23:35:33 +08:00
arkohut
b48f0f4a89 feat(library): add multiple folders at a time 2024-07-14 23:35:05 +08:00
arkohut
7212a2f65b feat: support only trigger webhook 2024-07-12 00:43:48 +08:00
arkohut
38272c48d0 feat(indexing): add api fetch document in typesense 2024-07-11 22:28:21 +08:00
arkohut
b5d524f7b0 feat: support scan and trigger specific plugins 2024-07-10 18:35:44 +08:00
arkohut
d1b83571e6 fix(scan): update timeout 2024-07-10 16:24:37 +08:00
arkohut
f8b5eb14e6 fix: add timeout when update entity 2024-07-07 00:02:39 +08:00
arkohut
0b03b2c860 refactor: use relative import 2024-07-04 00:28:45 +08:00
arkohut
fff57e607a fix: use relative path for web assets 2024-07-03 18:22:49 +08:00
arkohut
9ac7c4b5a5 feat: force update files 2024-07-03 00:02:41 +08:00
arkohut
d730d6c469 feat: change default limit to 48 2024-07-02 19:03:46 +08:00
arkohut
4426602393 feat: integrate svelte as spa 2024-07-02 17:23:12 +08:00
arkohut
facf05117b feat(server): add search and file api 2024-07-02 00:27:19 +08:00
arkohut
973c5e5006 feat(typesense): add auto embedding 2024-07-02 00:27:00 +08:00
arkohut
6d5d11bf77 feat: support -h for help 2024-06-23 13:46:25 +08:00
arkohut
8a1e04f997 feat: add index 2024-06-21 18:47:01 +08:00
arkohut
7c3017f85b feat: add tags for api 2024-06-13 23:39:57 +08:00
arkohut
d1575d55f8 feat: indexing entity to typesense 2024-06-13 23:32:55 +08:00
arkohut
7734b40848 fix(entity): entity update issues 2024-06-13 11:15:27 +08:00
arkohut
d6a9241ffc feat: use pydantic settings 2024-06-12 23:38:38 +08:00
arkohut
2ffd4b0b78 refactor: use magika for file type detect 2024-06-12 20:26:24 +08:00
arkohut
1dc7341b26 feat: ignore ds_store in mac 2024-06-12 18:57:31 +08:00
arkohut
b0becf64ea refactor(entity): change metadata data type to text, json etc 2024-06-11 15:18:00 +08:00
arkohut
107f7d06c2 refactor(entity): make entity a root resource 2024-06-11 14:24:17 +08:00
arkohut
9d6977ac4f test: dry 2024-06-11 13:25:58 +08:00
arkohut
354bf4f742 feat(entity): add update tag and metadata's api 2024-06-10 14:21:16 +08:00
arkohut
451c9f4af2 fix: trigger_webhooks_flag=true when scan update entity 2024-06-10 13:46:21 +08:00
arkohut
8ec89219f3 feat(entity): update tags and metadata 2024-06-10 00:29:27 +08:00
arkohut
0db275fba0 test: missing attr when update entity 2024-06-09 00:30:29 +08:00
arkohut
885629bf15 feat(entity): trigger webhooks when entity update or created 2024-06-07 23:13:01 +08:00
arkohut
a4d0ccc34e feat(library): show plugins 2024-06-07 19:55:45 +08:00
arkohut
adb4c2f845 fix(plugin): wrong mapping with lib 2024-06-07 19:52:55 +08:00
arkohut
649ed434c7 feat(plugin): bind lib and plugin 2024-06-07 19:30:15 +08:00
arkohut
64dd3a50f1 feat(plugin): create and list plugins 2024-06-07 19:22:06 +08:00
arkohut
8b77d823d1 feat(plugin): 400 when add duplicated plugin name 2024-06-07 19:03:55 +08:00
arkohut
22fc6a4e27 test(plugin): add create plugin test 2024-06-07 19:02:12 +08:00
arkohut
9e4e23469f feat: switch to absolute path 2024-06-07 13:22:17 +08:00
arkohut
cd3672afe7 refactor: use status.xx instead of number status code 2024-06-05 01:48:26 +08:00
arkohut
fe076da948 feat: prevent duplicated library name 2024-06-04 17:10:42 +08:00
arkohut
e36c0a6bec feat(folder): prevent duplicated folders 2024-06-04 17:03:31 +08:00
arkohut
e4fa1693d9 refactor: extract global endpoint 2024-06-04 16:48:44 +08:00
arkohut
981793165d feat: delete entities if files deleted 2024-06-04 16:43:07 +08:00
arkohut
bf6ec4ac02 feat(entity): delete entity 2024-06-04 16:36:33 +08:00
arkohut
96b219115d feat(folder): list entities in folder 2024-06-04 16:20:35 +08:00
arkohut
890245d654 refactor: use by-filepath instead of entities 2024-06-04 16:01:27 +08:00
arkohut
9f3d0819f7 feat(library): update entity when file updated 2024-06-04 15:19:09 +08:00
arkohut
20831d20ed feat(cli): scan library 2024-06-04 00:53:22 +08:00
arkohut
abf04fe74e feat(library): get library by id 2024-06-03 23:10:13 +08:00
arkohut
54091c94d4 feat(entity): get entity by id or filepath 2024-06-02 23:31:28 +08:00
arkohut
7512dacd52 feat(entity): create and update entity by id 2024-06-02 17:30:56 +08:00
arkohut
c1daee5f3d feat(cli): create library 2024-06-02 00:44:14 +08:00
arkohut
8caa5c5a7e feat(cli): ls libraries 2024-06-02 00:25:02 +08:00
arkohut
dd3b32821d feat: list libraries 2024-06-02 00:11:14 +08:00
arkohut
5c850a56c1 test: add first test 2024-06-01 13:46:30 +08:00
arkohut
ec03b818f3 feat: create and bind plugin to library 2024-06-01 13:13:35 +08:00
arkohut
121c8c51b1 feat: add entity 2024-06-01 13:03:53 +08:00
arkohut
929b81ff41 feat: update schemas 2024-06-01 12:52:42 +08:00
arkohut
7dc205ef4f feat(db): update db orm classes 2024-05-31 18:06:09 +08:00
arkohut
3d7cbe03b4 feat(db): add initial db schema 2024-05-31 00:16:14 +08:00
arkohut
e8a7a75d89 feat(library): library mock crud 2024-05-28 10:03:34 +08:00
arkohut
fc72c968d6 init commit 2024-05-27 10:35:20 +08:00