An account that has 8 badges, is returning only 1 incorrectly.

Example Tibia Character:
https://www.tibia.com/community/?subtopic=characters&name=Luminals

URL that has an error:
https://api.tibiadata.com/v3/character/Luminals

"account_badges": [
	{
		"name": "Veteran Hero&#39;, &#39;The account is older than 5 years.&#39;, &#39;&#39;);\" onmouseout=\"$(&#39;#HelperDivContainer&#39;).hide();\"><img src=\"https://static.tibia.com/images//badges/badge_accountage2.png\" alt=\"Veteran Hero\"/></span></span><span style=\"position: relative; top: 3px; margin: 10px;\"><span class=\"HelperDivIndicator\" onmouseover=\"ActivateHelperDiv($(this), &#39;Fledgeling Hero&#39;, &#39;The account is older than 1 year.&#39;, &#39;&#39;);\" onmouseout=\"$(&#39;#HelperDivContainer&#39;).hide();\"><img src=\"https://static.tibia.com/images//badges/badge_accountage1.png\" alt=\"Fledgeling Hero\"/></span></span><span style=\"position: relative; top: 3px; margin: 10px;\"><span class=\"HelperDivIndicator\" onmouseover=\"ActivateHelperDiv($(this), &#39;Tibia Loyalist (Grade 2)&#39;, &#39;The account earned more than 1000 loyalty points.&#39;, &#39;&#39;);\" onmouseout=\"$(&#39;#HelperDivContainer&#39;).hide();\"><img src=\"https://static.tibia.com/images//badges/badge_tibialoyalist2.png\" alt=\"Tibia Loyalist (Grade 2)\"/></span></span><span style=\"position: relative; top: 3px; margin: 10px;\"><span class=\"HelperDivIndicator\" onmouseover=\"ActivateHelperDiv($(this), &#39;Tibia Loyalist (Grade 1)&#39;, &#39;The account earned more than 100 loyalty points.&#39;, &#39;&#39;);\" onmouseout=\"$(&#39;#HelperDivContainer&#39;).hide();\"><img src=\"https://static.tibia.com/images//badges/badge_tibialoyalist1.png\" alt=\"Tibia Loyalist (Grade 1)\"/></span></span><span style=\"position: relative; top: 3px; margin: 10px;\"><span class=\"HelperDivIndicator\" onmouseover=\"ActivateHelperDiv($(this), &#39;Global Player (Grade 3)&#39;, &#39;Summing up the levels of all characters on the account amounts to at least 2000.&#39;, &#39;&#39;);\" onmouseout=\"$(&#39;#HelperDivContainer&#39;).hide();\"><img src=\"https://static.tibia.com/images//badges/badge_globalplayer3.png\" alt=\"Global Player (Grade 3)\"/></span></span><span style=\"position: relative; top: 3px; margin: 10px;\"><span class=\"HelperDivIndicator\" onmouseover=\"ActivateHelperDiv($(this), &#39;Global Player (Grade 2)&#39;, &#39;Summing up the levels of all characters on the account amounts to at least 1000.&#39;, &#39;&#39;);\" onmouseout=\"$(&#39;#HelperDivContainer&#39;).hide();\"><img src=\"https://static.tibia.com/images//badges/badge_globalplayer2.png\" alt=\"Global Player (Grade 2)\"/></span></span><span style=\"position: relative; top: 3px; margin: 10px;\"><span class=\"HelperDivIndicator\" onmouseover=\"ActivateHelperDiv($(this), &#39;Global Player (Grade 1)&#39;, &#39;Summing up the levels of all characters on the account amounts to at least 500.&#39;, &#39;&#39;);\" onmouseout=\"$(&#39;#HelperDivContainer&#39;).hide();\"><img src=\"https://static.tibia.com/images//badges/badge_globalplayer1.png\" alt=\"Global Player (Grade 1)\"/></span></span><span style=\"position: relative; top: 3px; margin: 10px;\"><span class=\"HelperDivIndicator\" onmouseover=\"ActivateHelperDiv($(this), &#39;Master Class (Grade 1)",
		"icon_url": "https://static.tibia.com/images//badges/badge_masterclass1.png",
		"description": "The account has reached at least level 100 with all four vocations."
	}
],

Url: https://api.tibiadata.com/v3/world/Honbra

{
    "worlds": {
        "world": {
            "name": "Honbra",
            "status": "online",
            "players_online": 145,
            "record_players": 1051,
            "record_date": "2017-07-03T02:20:31Z",
            "creation_date": "0001-01-01",
            "location": "South America",
            "pvp_type": "Open PvP",
            "premium_only": false,
            "transfer_type": "",
            "world_quest_titles": [...],
            "battleye_protected": true,
            "battleye_date": "release",
            "game_world_type": "regular",
            "tournament_world_type": "",
            "online_players": [...]
        }
    },
    "information": {
        "api_version": 3,
        "timestamp": "2023-08-08T17:03:02Z"
    }
}

Creation date with invalid value: "creation_date": "0001-01-01",

Seems like the TibiadataDatetimeV3 doesn't take the time zone in to account when parsing content.
Tests are therefore failing when running locally, but work on GitHub.

Maybe we should add some indicator that the deaths list is truncated as well..
deaths_truncated or something maybe?

Originally posted by @tobiasehlert in #272 (comment)

Add two new endpoints to application so that kubernetes can monitor it.

• GET /healthz (replacing /health)
• GET /readyz

There seems to be an issue with the title parsing of characters.

Current response:

{
    "character": {
        "character": {
            "name": "Nocna Furia",
            "sex": "male",
            "title": "Tibia's Topmodel",
            "unlocked_titles": 0,
...

Expected response:

{
    "character": {
        "character": {
            "name": "Nocna Furia",
            "sex": "male",
            "title": "Tibia's Topmodel (Grade 1)",
            "unlocked_titles": 6,
...

URL:

• https://dev.tibiadata.com/v4/character/Nocna%20Furia

Error from log:

2023/10/08 19:58:30 [warning] TibiaDataStringToInteger: couldn't convert string into int. error: strconv.Atoi: parsing "Grade": invalid syntax
[GIN] 2023/10/08 - 19:58:30 | 200 |  141.134276ms |   XXX.XXX.XXX.XXX | GET      "/v4/character/Nocna Furia"

When codecov workflow executes, it throws 3 errors saying it receives Bad Request from upstream.

=== RUN   TestFakeToUpCodeCoverage
Warning: 2023/10/09 11:23:24 [warning] TibiaCreaturesCreatureImpl called on invalid creature
    webserver_test.go:65: 
        	Error Trace:	/home/runner/work/tibiadata-api-go/tibiadata-api-go/src/webserver_test.go:65
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 400
        	Test:       	TestFakeToUpCodeCoverage
    webserver_test.go:85: 
        	Error Trace:	/home/runner/work/tibiadata-api-go/tibiadata-api-go/src/webserver_test.go:85
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 400
        	Test:       	TestFakeToUpCodeCoverage
    webserver_test.go:124: 
        	Error Trace:	/home/runner/work/tibiadata-api-go/tibiadata-api-go/src/webserver_test.go:124
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 400
        	Test:       	TestFakeToUpCodeCoverage
--- FAIL: TestFakeToUpCodeCoverage (1.86s)

When adding print of c (c, _ = gin.CreateTestContext(w)) and w (w = httptest.NewRecorder()) it looks like this..

tibiaCreaturesCreature

c:
&{{0xc000084c40 174 400} <nil> 0xc00012c200 [{race demon}] [] -1  0xc000082d00 0xc0000b85e8 0xc0000b8600 {{0 0} 0 0 {{} 0} {{} 0}} map[]  [] map[] map[] 0}

w:
&{400 map[Content-Type:[application/json; charset=utf-8]] {"information":{"api":{"version":0,"release":"","commit":""},"timestamp":"2023-10-09T12:26:42Z","status":{"http_code":400,"error":20002,"message":"could not find creature"}}} false <nil> map[Content-Type:[application/json; charset=utf-8]] true}

tibiaGuildsGuild

c:
&{{0xc000084f80 171 400} <nil> 0xc0004c0f00 [{name pax}] [] -1  0xc0005a31e0 0xc00036ffe0 0xc0004ea000 {{0 0} 0 0 {{} 0} {{} 0}} map[]  [] map[] map[] 0}

w:
&{400 map[Content-Type:[application/json; charset=utf-8]] {"information":{"api":{"version":0,"release":"","commit":""},"timestamp":"2023-10-09T12:26:42Z","status":{"http_code":400,"error":20004,"message":"could not find guild"}}} false <nil> map[Content-Type:[application/json; charset=utf-8]] true}

tibiaHighscores

c:
&{{0xc000084b00 203 400} <nil> 0xc00012c400 [{world antica} {category experience} {vocation sorcerer} {page 4}] [] -1  0xc000082d00 0xc0004ea228 0xc0004ea240 {{0 0} 0 0 {{} 0} {{} 0}} map[]  [] map[] map[] 0}

w:
&{400 map[Content-Type:[application/json; charset=utf-8]] {"information":{"api":{"version":0,"release":"","commit":""},"timestamp":"2023-10-09T12:26:42Z","status":{"http_code":400,"error":11008,"message":"the provided page is larger than max amount of pages"}}} false <nil> map[Content-Type:[application/json; charset=utf-8]] true}

Example URL:
https://api.tibiadata.com/v3/fansites

Currently:

{
    "name": "TibiaData.com",
    "languages": [],
    ...
}

Expected:

{
    "name": "TibiaData.com",
    "languages": [ "en" ],
    ...
}

I am receiving the error response with http code 400 when I try to access the following endpoint:
https://api.tibiadata.com/v3/highscores/Alumbra/magiclevel/all/22

The response says:

{"error":"page needs to be from 1 to 20"}

Actually, Alumbra magic level highscores have 22 pages and the endpoint also says:

"highscore_page":{"current_page":1,"total_pages":22,"total_records":1060}

reported by email

rel #162

Old names can be pretty long so we need to increase it slightly.

Example:

• Spiritbreaker'Strong
• Marizinhahh'Dragon

URL:

• https://dev.tibiadata.com/v4/character/Spiritbreaker'Strong

For example, https://api.tibiadata.com/v3/news/id/504 returns the following (after pretty-printing):

{
  "news": {
    "id": 504,
    "date": "2007-04-27",
    "category": "community",
    "type": "ticker",
    "url": "https://www.tibia.com/news/?subtopic=newsarchive&id=504",
    "content": "",
    "content_html": ""
  },
  "information": {
    "api_version": 3,
    "timestamp": "2023-11-09T12:10:50Z"
  }
}

However, the news ticker at https://www.tibia.com/news/?subtopic=newsarchive&id=504 does have content:

A new feedback form has been released today. Help us to find out which websites and magazines are popular in your country by filling out the new questionnaire. In our current poll we are curious about your occupation.

Here’s some example affected IDs (these might all share the same root cause):

444
445
447
448
451
453
454
455
456
458
459
460
463
464
465
466
467
469
470
471
474
475
476
477
478
481
482
483
484
485
486
487
489
490
491
492
493
494
497
498
500
501
503
504
506
507
509
510
512
514
516
517
518
520
522
523
524
527
529
530
532
533
535
537
539
540
541
544
545
546
548
549
550
552
554
556
557
559
560
563
564
567
568
569
570
572
574
575
576
577
578
579
580
581
582
588
589
590
591
592
594
595
596
599
600
603
604
608
609
610
612
613
615
616
617
618
621
622
625
626
627
628
629
631
635
636
637
639
640
641
642
644
645
648
649
651
652
653
654
655
656
657
658
660
661
662
664
665
666
669
670
675
676
677
678
679
681
682
683
684
686
687
688
689
690
692
693
694
695
696
697
701
702
703
704
705
706
709
710
711
712
713
715
717
718
720
721
722
723
725
726
727
729
730
731
733
735
736
737
739
740
742
743
744
745
746
749
751
752
757
758
759
760
763
764
765
768
769
770
771
773
774
775
776
778
779
780
781
782
783
786
787
788
789
790
792
793
794
795
796
797
798
800
801
803
806
807
808
809
812
813
814
816
817
818
820
821
823
825
826
827
829
831
832
833
834
837
838
839
841
842
843
846
847
848
849
851
852
853
854
856
857
858
862
866
867
868
871
875
877
879
880
881
882
885
886
889
891
892
893
895
896
898
899
900
902
903
904
905
907
908
911
912
913
914
915
917
919
920
922
923
924
925
926
927
929
930
931
932
933
935
936
938
939
940
941
942
943
947
949
950
952
953
954
956
958
959
961
962
963
964
966
967
968
970
972
973
974
975
976
977
979
981
982
983
985
986
987
989
990
991
992
993
997
999
1000
1001
1004
1005
1006
1007
1010
1011
1012
1013
1015
1016
1018
1019
1020
1022
1024
1025
1026
1028
1031
1032
1033
1034
1035
1037
1038
1039
1041
1042
1043
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1058
1059
1060
1061
1062
1063
1064
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1078
1079
1080
1081
1082
1084
1085
1086
1087
1088
1090
1091
1092
1093
1094
1096
(and many more)

Need support for double-quotes in spell formula

URL:
https://dev.tibiadata.com/v3/spells/spell/levitate

Character Mieluffy
Jan 12 2024, 20:38:09 CET Eliminated at Level 508 by [Shantty], [Sin Primalbazzar], [Supperiore], [Righo], [Beltran Tzawayak], [Ilysz Sin Tales], [Flaco El Pobre], [Pecaas], [Knightsitaz], [Dhanielz Acorazado], [Jobi], [Aeronabic], [Side Effectss], [Ekizdd], [Baby Mikoh], [Sneki], Love and Death (traded) and [Next Generation].
Assisted by [Swiifti] and [Duende blanco].

It gets parsed to:

{
  "name": "Love",
  "player": false,
  "traded": false,
  "summon": ""
},
{
  "name": "Death",
  "player": true,
  "traded": true,
  "summon": ""
}

So the character's name "Love and Death" is wrongfully split into two parts, with the first part being mistaken for the creature's name. A fix should be simple: a creature can only appear last in the killers list. So even if the traded character appears last, then what is before "and" and without a hyperlink is part of his name anyway, not a creature name. I would write a fix but I don't know Golang (yet?)

The view of a single house/guildhall does now contain a string based on the actual house type and not based on the injected GET value.
Therefore the house type doesn't need to be loaded from the assets file (by tibiadata-api-assets).

1. Request /v4/world/{name} with not existing world return status code 400(Bad Request) instead of 404(Not Found)
   e.g. "https://api.tibiadata.com/v4/world/Anticaa"
   return:

{
  "information": {
    "api": {
      "version": 4,
      "release": "4.0.2",
      "commit": "ced9b5f8c1929c079756998f37a0f560bc643c2d"
    },
    "timestamp": "2024-01-09T11:09:50Z",
    "status": {
      "http_code": 400,
      "error": 11002,
      "message": "the provided world does not exist"
    }
  }
}

2. Request /v4/character/{name} with not existing character return status code 502(Bad Gateway) instead of 404(Not Found)
   e.g. "https://api.tibiadata.com/v4/character/Trollefara"
   return:

{
  "information": {
    "api": {
      "version": 4,
      "release": "4.0.2",
      "commit": "ced9b5f8c1929c079756998f37a0f560bc643c2d"
    },
    "timestamp": "2024-01-09T11:11:31Z",
    "status": {
      "http_code": 502,
      "error": 20001,
      "message": "could not find character"
    }
  }
}

This happens to normal monsters who have ' of ' in their name and prefixed with 'a' or 'an' in the death message which causes them to match against summonRegex. A few examples would be sight of surrender, spark of destruction and the X of the cult family. A simple solution at a glance seems to be moving the get summon block inside TibiaDataParseKiller into the already existing if player conditional branch, and I don't believe this would affect deaths by summons of monsters since as far as I know they're always listed separately from their owner.

Problem

Not all kills shown on tibia.com appear in the deathlog in tibiadata

Expected behavior

all kills are shown, even if some fields cannot be populated

Detail

Any death without any identifiable killer that only has "assists" is omitted from the death list. These deaths can occur when for example a character is killed only by members of their own guild. Presumably these kills are omitted because not all information is available, (Level the death occurred at etc).

I think these deaths should still appear in the deathlist for the character, otherwise this deathlist does not accurately reflect what happened.

Examples

Characters which (currently) serve as examples:

• Sir Pixxy
• Suedi Baaz

Extra Background

Deaths of this kind are common on retro-hardcore servers where kill selling is widespread, as being in the same guild as the kill seller reduces protection zone block so buyers do not have to wait the full 15 minutes.

Date format has changed on world pages.

Old format:
10/17

New format:
October 2017

Impacts on:

• /v4/world/{name} #240
• /v3/world/{name} #241

Creatures endpoint use the race parameter as provided in the API URL.

Example URL:
https://dev.tibiadata.com/v4/creature/DeMonS

Issue:

{
  "creature": {
    "name": "Demons",
    "race": "DeMonS",
    "image_url": "https://static.tibia.com/images/library/demon.gif",
    ...

Expected:

{
  "creature": {
    "name": "Demons",
    "race": "demon",
    "image_url": "https://static.tibia.com/images/library/demon.gif",
    ...

Following if statement will also not evaluate correct, when race is not exact string as in parsed HTML.

Suggestion:
We should probably use the race value from the static tibia image link to compare with.

There seems to be an issue with the parsing of the timestamps in deaths on character endpoint.

...
    "deaths": [
        {
            "time": "0001-01-01T00:00:00Z",
            "level": 10,
            "killers": [
                {
                    "name": "fire",
                    "player": false,
                    "traded": false,
                    "summon": ""
                }
            ],
            "assists": [],
            "reason": "t 08 2023, 18:19:35 CESTDied at Level 10 by fire."
        },
        ...
    ]
...

URL:

• https://dev.tibiadata.com/v4/character/Orca%20Kaoksh

Error from log:

2023/10/08 19:47:08 parsing time "Oc" as "Jan 02 2006, 15:04:05 MST": cannot parse "Oc" as "Jan"
2023/10/08 19:47:08 parsing time "Oc" as "Jan 02 2006, 15:04:05 MST": cannot parse "Oc" as "Jan"
[GIN] 2023/10/08 - 19:47:08 | 200 |  108.456648ms |  XXX.XXX.XXX.XXX | GET      "/v4/character/Orca Kaoksh"

https://api.tibiadata.com/v4/killstatistics/Ambra returns

{
  "information": {
    "api": {
      "version": 4,
      "release": "edge",
      "commit": "209a9cab41f4f0d80ec00ae9589894a9c41e9baa"
    },
    "timestamp": "2023-10-19T11:48:43Z",
    "status": {
      "http_code": 400,
      "error": 11002,
      "message": "the provided world does not exist"
    }
  }
}

even though https://www.tibia.com/community/?subtopic=killstatistics&world=Ambra shows data.

The same goes for the /v4/world/* endpoint, e.g. https://api.tibiadata.com/v4/world/Ambra.

In general I’d expect all world names returned in https://api.tibiadata.com/v4/worlds to have functioning corresponding /killstatistics/* and /world/* endpoints.

The applications panics when character has too many deaths.

Last line of deaths table shows this:
There happened more character deaths in the last 30 days than we can display here.

URL:

• https://dev.tibiadata.com/v4/character/Stalone%20Matador

Error from log:

2023/10/09 20:27:23 [warning] TibiaDataStringToInteger: couldn't convert string into int. error: strconv.Atoi: parsing "ppened": invalid syntax

[GIN] 2023/10/09 - 20:27:23 | 200 |   104.78069ms |   10.233.86.153 | GET      "/v4/character/Stalone Matador"

2023/10/09 20:27:23 [Recovery] 2023/10/09 - 20:27:23 panic recovered:
runtime error: slice bounds out of range [2:1]
/usr/local/go/src/runtime/panic.go:154 (0x433edb)
/go/src/src/TibiaCharactersCharacter.go:446 (0x7bb8f0)
/go/pkg/mod/github.com/!puerkito!bio/[email protected]/iteration.go:21 (0x6e0bb4)
/go/src/src/TibiaCharactersCharacter.go:375 (0x7b9fa6)
/go/pkg/mod/github.com/!puerkito!bio/[email protected]/iteration.go:21 (0x6e0bb4)
/go/src/src/TibiaCharactersCharacter.go:151 (0x7b97d8)
/go/src/src/webserver.go:310 (0x7d9037)
/go/src/src/webserver.go:1118 (0x7d6ffb)
/go/src/src/webserver.go:306 (0x7d2550)
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 (0x7b7e65)
/go/pkg/mod/github.com/gin-contrib/[email protected]/handler.go:60 (0x7b7e47)
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 (0x7adad9)
/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102 (0x7adac7)
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 (0x7acc7d)
/go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:240 (0x7acc4c)
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 (0x7abd3a)
/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620 (0x7ab9cd)
/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576 (0x7ab67c)
/usr/local/go/src/net/http/server.go:2938 (0x64874d)
/usr/local/go/src/net/http/server.go:2009 (0x645693)
/usr/local/go/src/runtime/asm_amd64.s:1650 (0x468240)

tibia.com doesn't return more than 50 records per page, which was previously possible by setting currentpage to 400000000000000

we'd need pagination added to resolve this issue.

There is an issue with town name validation when towns have a space in the name.

failing url:

• https://api.tibiadata.com/v4/houses/Ambra/Gray+Beach
• https://api.tibiadata.com/v4/houses/Ambra/Liberty+Bay
• https://api.tibiadata.com/v4/houses/Ambra/Port+Hope

working url:

• https://api.tibiadata.com/v4/houses/Ambra/Gray Beach
• https://api.tibiadata.com/v4/houses/Ambra/Liberty Bay
• https://api.tibiadata.com/v4/houses/Ambra/Port Hope

response on failing url:

{
    "information": {
        "api": {
            "version": 4,
            "release": "edge",
            "commit": "39226677b2c7582c91e70cb367c33ef7763cf4ac"
        },
        "timestamp": "2023-11-23T09:00:58Z",
        "status": {
            "http_code": 400,
            "error": 11006,
            "message": "the provided town does not exist"
        }
    }
}

Have you considered exposing an endpoint for getting the Drome leaderboard data? Parameters would be world and rotation, where rotation would default to the current one, just like on the website:

The source of truth would be, e.g.

https://www.tibia.com/community/?subtopic=leaderboards&world=Vunira&rotation=12

…for the Vunira leaderboard of Drome rotation 12.

Something seems to differ between v3 and v4 on town Ab'Dendriel.

failing url:

• https://api.tibiadata.com/v4/houses/Antica/Ab'Dendriel

working url:

• https://api.tibiadata.com/v3/houses/Antica/Ab'Dendriel

response on failing url:

{
    "houses": {
        "world": "Ambra",
        "town": "Ab'dendriel",
        "house_list": null,
        "guildhall_list": null
    },
    "information": {
        "api": {
            "version": 4,
            "release": "edge",
            "commit": "cfecd592a1f9372876cab270c61242fde5a3c2dc"
        },
        "timestamp": "2023-11-24T21:00:10Z",
        "status": {
            "http_code": 200
        }
    }
}

Doesn't seem to be related to changes made in #294.

For some combinations of server, voc, category, there are more than 20 pages to a highscores list (e.g. antica has 40 pages for knight magic level). But when I request some of these later pages, the API does not give me results.

e.g. on v3, any page above 23 doesn't give results: https://api.tibiadata.com/v3/highscores/Antica/magiclevel/knight/24
And on v4, any page above 30 doesn't give results: https://dev.tibiadata.com/v4/highscores/Antica/magiclevel/knight/31
This is despite both of these endpoints returning total_pages: 40
I am able to browse these high page numbers without a problem on the tibia website

The json payload returned by some calls is redundant, for example:

/v4/character/<name>

# Before
{
  "character": {
    "character": {
    ...
    }
}

# After
{
  "data": {
    "character": {
    ...
    }
}

/v4/boostablebosses

# Before
{
  "boostable_bosses": {
    "boostable_boss_list": [
    ]
}

# After
{
  "data": {
    "boostable_bosses": [
    ]
}

It makes more sense to always return a data field with the data we are expecting. This is a step towards standardization of the api responses.

In addition, the information field is irrelevant in most of the cases, the status header would be enough for status code and the api information could have a dedicated endpoint if we want to retrieve this information.

Example URL that returns zero content and generated error in log:
https://dev.tibiadata.com/v3/character/Abandonado

Log output:

2022/02/28 08:50:29 [Recovery] 2022/02/28 - 08:50:29 panic recovered:
runtime error: index out of range [0] with length 0
/usr/local/go/src/runtime/panic.go:90 (0x432734)
        goPanicIndex: panic(boundsError{x: int64(x), signed: true, y: y, code: boundsIndex})
/mnt/c/Users/TOLI/GIT/tibiadata-api-go/src/TibiaCharactersCharacterV3.go:276 (0x927791)
        TibiaCharactersCharacterV3Impl.func1.2: Name:        subma1[0][1],
/home/tobias/go/pkg/mod/github.com/!puerkito!bio/goquery@v1.8.0/iteration.go:10 (0x72ed45)
        (*Selection).Each: f(i, newSingleSelection(n, s.document))
/mnt/c/Users/TOLI/GIT/tibiadata-api-go/src/TibiaCharactersCharacterV3.go:261 (0x925f9a)
        TibiaCharactersCharacterV3Impl.func1: CharacterDivQuery.Find(".TableContentContainer tr td").Each(func(index int, s *goquery.Selection) {      
/home/tobias/go/pkg/mod/github.com/!puerkito!bio/goquery@v1.8.0/iteration.go:10 (0x72ed45)
        (*Selection).Each: f(i, newSingleSelection(n, s.document))
/mnt/c/Users/TOLI/GIT/tibiadata-api-go/src/TibiaCharactersCharacterV3.go:153 (0x925ad8)
        TibiaCharactersCharacterV3Impl: ReaderHTML.Find(".TableContainer").Each(func(index int, s *goquery.Selection) {
/mnt/c/Users/TOLI/GIT/tibiadata-api-go/src/webserver.go:194 (0x93c6d1)
        tibiaCharactersCharacterV3.func1: return TibiaCharactersCharacterV3Impl(BoxContentHTML), http.StatusOK
/mnt/c/Users/TOLI/GIT/tibiadata-api-go/src/webserver.go:724 (0x93aa14)
        tibiaDataRequestHandler: jsonData, httpStatusCode := requestHandler(BoxContentHTML)
/mnt/c/Users/TOLI/GIT/tibiadata-api-go/src/webserver.go:190 (0x937d95)
        tibiaCharactersCharacterV3: tibiaDataRequestHandler(
/home/tobias/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/context.go:168 (0x915321)
        (*Context).Next: c.handlers[c.index](c)
/home/tobias/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/recovery.go:99 (0x91530c)
        CustomRecoveryWithWriter.func1: c.Next()
/home/tobias/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/context.go:168 (0x914586)
        (*Context).Next: c.handlers[c.index](c)
/home/tobias/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/logger.go:241 (0x914569)
        LoggerWithConfig.func1: c.Next()
/home/tobias/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/context.go:168 (0x913ad0)
        (*Context).Next: c.handlers[c.index](c)
/home/tobias/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/gin.go:555 (0x913738)
        (*Engine).handleHTTPRequest: c.Next()
/home/tobias/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/gin.go:511 (0x913271)
        (*Engine).ServeHTTP: engine.handleHTTPRequest(c)
/usr/local/go/src/net/http/server.go:2879 (0x69d37a)
        serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
/usr/local/go/src/net/http/server.go:1930 (0x698a27)
        (*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
/usr/local/go/src/runtime/asm_amd64.s:1581 (0x465980)
        goexit: BYTE    $0x90   // NOP

Spec should response should be:

{
"character": {
"account_badges": [
{
"description": "string",
"icon_url": "string",
"name": "string"
}],
"account_information": {
"created": "string",
"loyalty_title": "string",
"position": "string"
},

Actual response:

{
  "character": {
    "character": {
      "name": "Trollefar",
      "sex": "male",
      "title": "Trolltrasher",
      "unlocked_titles": 7,
      "vocation": "Elite Knight",

There is an extra object character in the actual response.

Also, I noticed some /v4/character broke lookup of some name such as Crocodilo'Dundee returns 400 with "message": "the provided character name has a word too big"

Currently, the only way to get a list of all houses and guildhalls in the game is to query

https://api.tibiadata.com/v3/houses/Antica/Venore

…replacing Venore with every city that has buildings for rent.

It would be lovely if either

https://api.tibiadata.com/v3/houses/Antica

…would give the complete list of buildings across all cities.

Alternatively (and I suspect you’ll prefer this option), it would be great if there was an API endpoint that returned all the cities with buildings, as listed in the Towns column on this page: https://www.tibia.com/community/?subtopic=houses

Currently, this list is:

Ab'Dendriel
Ankrahmun
Carlin
Darashia
Edron
Farmine
Gray Beach
Issavi
Kazordoon
Liberty Bay
Moonfall
Port Hope
Rathleton
Silvertides
Svargrond
Thais
Venore
Yalahar

…but any update can potentially change it.

Example URL:
https://api.tibiadata.com/v3/character/Trollefar

Currently:

"guild": {
  "name": "Elysium",
  "rank": "Follower of the "
}

Expected:

"guild": {
  "name": "Elysium",
  "rank": "Follower"
}

Some creatures are healed by elemental damage. tibia library displays this information.

What is more, they are healed when fire damage is used against them.

source: https://www.tibia.com/library/?subtopic=creatures&race=lavablob

I would expect this information to be in a [string] field and work the same as immune, strong, and weakness fields

This data could be scraped from https://www.tibia.com/library/?subtopic=creatures and https://www.tibia.com/library/?subtopic=boostablebosses, respectively, or in a single request by visiting most pages on tibia.com e.g. https://www.tibia.com/news/?subtopic=latestnews where the HTML contains something like:

<img id="Pedestal" src="https://static.tibia.com/images/global/header/pedestal.gif" alt="Monster Pedestal Box" /><br><img id="Monster" title="Today's boosted creature: Gozzler" src="https://static.tibia.com/images/global/header/monsters/gozzler.gif" onClick="window.location = 'https://www.tibia.com/library/?subtopic=creatures';" alt="Boosted Creature" /><img id="Boss" title="Today's boosted boss: Srezz Yellow Eyes" src="https://static.tibia.com/images/global/header/monsters/srezz.gif" onClick="window.location = 'https://www.tibia.com/library/?subtopic=boostablebosses';" alt="Boosted Boss" />

We need to have some way to detect maintenance mode and return some proper message.
This should be done in the TibiadataHTMLDataCollectorV3 function or somewhere more suitable.

Here is some initial logging at least:

There is problem with nicknames with apostrophes [ ' ], instead of apostroph there is \u0026#39;

The problems found in:

/v3/guild/...

{"name":"Alex\u0026#39;Ifrit","title":"","rank":"Member","vocation":"Sorcerer","level":41,"joined":"2022-01-16","status":"offline"}

not sure, but it might be also in other places where you import names of players, like:

/v3/world/...

cant really check, cause its hard to find ppl with apostrophe online 😂

/v3/character/...

might be in killers and reasons like in #97 with nbsp's

Multiple values seem to be returned empty or as null.

Actual response for Dark celebra:

{
  "guilds": {
    "guild": {
      "name": "Dark celebra",
      "world": "",
      "guildhalls": ,
      "founded": "",

Expected response for Dark celebra:

{
  "guilds": {
    "guild": {
      "name": "Dark celebra",
      "world": "Celebra",
      "guildhalls": [{
          "name": "Castle of the Winds",
          "paid_until": "2022-03-15",
          "world": "Celebra"
        }],
      "founded": "",

Example URL:
https://api.tibiadata.com/v3/guild/Dark+celebra

Guild name:
New era

Guild description:
New era Team, The guild was founded on Fortera on Mar 23 2012. It is currently active. Fortera siempre en mi corazón

Issue:
Description string contains The guild was founded on string.

Trace:

2022/07/21 08:14:28 [Recovery] 2022/07/21 - 08:14:28 panic recovered:
runtime error: index out of range [0] with length 0
/usr/local/go/src/runtime/panic.go:89 (0x43317e)
        goPanicIndex: panic(boundsError{x: int64(x), signed: true, y: y, code: boundsIndex})
/mnt/c/Users/TOLI/GIT/tibiadata-api-go/src/TibiaGuildsGuildV3.go:147 (0x7c2a04)
        TibiaGuildsGuildV3Impl: GuildWorld = subma1b[0][1]
/mnt/c/Users/TOLI/GIT/tibiadata-api-go/src/webserver.go:294 (0x7cdfa9)
        tibiaGuildsGuildV3.func1: return TibiaGuildsGuildV3Impl(guild, BoxContentHTML), http.StatusOK        
/mnt/c/Users/TOLI/GIT/tibiadata-api-go/src/webserver.go:724 (0x7d10f2)
        tibiaDataRequestHandler: jsonData, httpStatusCode := requestHandler(BoxContentHTML)
/mnt/c/Users/TOLI/GIT/tibiadata-api-go/src/webserver.go:290 (0x7cdeab)
        tibiaGuildsGuildV3: tibiaDataRequestHandler(
/home/tobias/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:173 (0x7aeee1)
        (*Context).Next: c.handlers[c.index](c)
/home/tobias/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:101 (0x7aeecc)
        CustomRecoveryWithWriter.func1: c.Next()
/home/tobias/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:173 (0x7adfc6)
        (*Context).Next: c.handlers[c.index](c)
/home/tobias/go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:240 (0x7adfa9)
        LoggerWithConfig.func1: c.Next()
/home/tobias/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:173 (0x7ad090)
        (*Context).Next: c.handlers[c.index](c)
/home/tobias/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:616 (0x7accf8)
        (*Engine).handleHTTPRequest: c.Next()
/home/tobias/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:572 (0x7ac9bc)
        (*Engine).ServeHTTP: engine.handleHTTPRequest(c)
/usr/local/go/src/net/http/server.go:2916 (0x65af9a)
        serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
/usr/local/go/src/net/http/server.go:1966 (0x657756)
        (*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
/usr/local/go/src/runtime/asm_amd64.s:1571 (0x465220)
        goexit: BYTE    $0x90   // NOP

[GIN] 2022/07/21 - 08:14:28 | 500 |    100.8026ms |       127.0.0.1 | GET      "/v3/guild/New+era"

Reported by @LAwade

Im pretty noob at this point, so if its not a bug - sorry.
Well, i rewrite my app to use v3 instead of v2 and after 3 days of using i realized that in every deaths, in every name with "space", instead of normal spaces, there is everywhere non-breaking spaces what literally fucked up everything in my case (for example i cant search for character name, becouse "xxx xxx" its not the same as "xxx\u00a0xxx")
well... i got 10k deaths in my db with that bullshit spaces... i belive that in v2 there are normal spaces, becouse i never had that problem before
the problem is with "reasons" and also with all the names separately at deathlist...
ofc i can fix it by my side while adding everything to db, but im pretty sure it should be changed at api side

I noticed some /v4/character broke lookup of some name such as Crocodilo'Dundee returns 400 with

{
  "information": {
    "api": {
      "version": 4,
      "release": "edge",
      "commit": "5a7fdbf4da820ab7492c71d1dafeea1b740abaaf"
    },
    "timestamp": "2023-02-19T06:36:26Z",
    "status": {
      "http_code": 400,
      "error": 10006,
      "message": "the provided character name has a word too big"
    }
  }
}

The same name works fine with v3

Character length is not calculated correct, when using + instead of spacing in character endpoint.

Issue:
Throws a 400 http code with error 10006:
the provided character name has a word too big

Example:

• https://api.tibiadata.com/v4/character/Jamil+Ondeta+Temconfusao
• https://api.tibiadata.com/v4/character/Traicionaron+Al+Mikee

Temporary workaround:
Ensure you replace + signs with space or %20 in your code.
E.g:

• https://api.tibiadata.com/v4/character/Jamil Ondeta Temconfusao
• https://api.tibiadata.com/v4/character/Jamil%20Ondeta%20Temconfusao

This is an easily worked around issue, but it doesn't exist in v3:
https://api.tibiadata.com/v3/character/Jamil+Ondeta+Temconfusao

So it may cause friction migrating, especially as some people may use + as spaces in their code-base due to habit as that is what the Tibia website uses.

Character name length check needs to be adjusted to support names with shorter names like Dragon Archer I.

Example:

• Dragon Archer I

URL:

• https://dev.tibiadata.com/v4/character/Dragon%20Archer%20I

We need to respond with proper error codes and messages.

Example scenarios:

• 404 if the resource is not found (e.g. faulty character name)
• 429 if the request is being throttled (tibia code is actually 403)
• 503 if there was some upstream issue
• and also something when tibia is having maintenance.

Section that needs to be rewritten/enhanced:

This should then also be added to the documentation annotation, so that docs get automatic update too.

TLDR

(Original TLDR at the end of this comment)

While I was working on removing regexes from parsers, I noticed that the speed gain and reduced usage was not as much as I expected it to be. So I decided to rework the parsers completely, instead of just removing regexes.

For more complex parsers, such as the character one, I'm gonna start by removing the regexes and later try to rewrite it completely. As this is a far more complex task and removing the regexes is already a huge performance boost for such parsers.

But for boostable bosses, for example, while removing regexes gave a 50% speed boost and reduced allocations by 4% rewriting the parser was relatively simple and gave a 98% speed boost while reducing allocations by almost 100%

Files to remove regex from or rewrite parsers:

TibiaBoostableBossesOverview

• #233

TibiaCharactersCharacter

• #223
• #224
• #225
• #226
• #227
• #228

TibiaCreaturesCreature

• CreatureDataRegex
• CreatureHitpointsRegex
• CreatureImmuneRegex
• CreatureStrongRegex
• CreatureWeakRegex
• CreatureHealedRegex
• CreatureManaRequiredRegex
• CreatureLootRegex

TibiaCreaturesOverview

• BoostedCreatureNameAndRaceRegex
• BoostedCreatureImageRegex
• CreatureInformationRegex

TibiaDataUtils

• #225
• removeUrlRegex

TibiaFansites

• FansiteInformationRegex
• FansiteImgTagRegex
• FansiteLanguagesRegex
• FansiteAnchorRegex

TibiaGuildsGuild

• GuildLogoRegex
• GuildWorldAndFoundationRegex
• GuildHomepageRegex
• GuildhallRegex
• GuildDisbaneRegex
• GuildMemberInformationRegex
• GuildMemberInvitesInformationRegex

TibiaHighscores

• HighscoresAgeRegex
• HighscoresPageRegex
• SevenColumnRegex
• SixColumnRegex

TibiaHousesHouse

• houseDataRegex
• housePassingRegex
• moveOutRegex
• paidUntilRegex
• houseAuctionedRegex

TibiaHousesOverview

• houseOverviewDataRegex
• houseOverviewAuctionedRegex

TibiaNews

• martelRegex

TibiaSpellsSpell

• SpellDataRowRegex
• SpellNameAndImageRegex
• SpellCooldownRegex
• SpellDescriptionRegex

TibiaWorldsOverview

• worldPlayerRecordRegex
• worldInformationRegex
• worldBattlEyeProtectedSinceRegex

TibiaWorldsWorld

• WorldDataRowRegex
• WorldRecordInformationRegex
• BattlEyeProtectedSinceRegex
• OnlinePlayerRegex

tibia

• characterNameRegex
• creatureAndSpellNameRegex
• guildNameRegex

I decided to open this issue in order to track the progress of removing all regex from the system.
Reducing the amount of regex is beneficial for two main reasons:

Speed


When parsing without regex, it is quicker


Heap Allocations


Even though we already made the regexes sentinel value, not having to compile them on every iteration, using the compiled regex still make heap allocations. So, by removing regexes completely we reduce a lot of heap allocations, thus reducing the GC load.

Creature "spark of destruction" is getting parsed wrong.

List of various creatures:
https://pastebin.com/9m4He1pq

URL that has an error:
https://api.tibiadata.com/v3/character/Lupi%20Lucky

{
	"time": "2022-05-08T00:26:31Z",
	"level": 537,
	"killers": [{
		"name": "destruction",
		"player": false,
		"traded": false,
		"summon": "a spark"
	}],
	"assists": [],
	"reason": "Died at Level 537 by a spark of destruction."
}

Issue with regex not handling singular of beds correctly.

Example of house with one bed:
https://api.tibiadata.com/v3/house/Belobra/10901

In TibiaGuildsOverviewV3, the description of a guild is not correct.
Since we run a removal of line breaks, but when commenting out line shown below, the whole endpoint fails.

Temporarily fixed endpoint with commit 5179ba9.

Example URLs to compare (search for guild Elysium):
https://dev.tibiadata.com/v3/guilds/world/Vunira
https://dev.tibiadata.com/v3/guilds/guild/Elysium

tibia.com uses latin-1 (ISO-8859-1) and we need some better converting method to be able to run only utf-8 through out the whole API.

This should probably be done right inside the TibiadataHTMLDataCollectorV3 function.

ONLY OCCURS DURING SERVER SAVE (10:00 CEST).

If any house in the list includes "finished", the API will respond with a blank page, or 404.
Requests towards the API will get 500 back.

Example links:
https://api.tibiadata.com/v3/houses/Antica/Ankrahmun
https://api.tibiadata.com/v3/houses/Nadora/Liberty%20Bay